Generating a secure random number by determining a change in parameters of digital content in subsequent frames via graphics processing circuitry

ABSTRACT

An apparatus, method, and computer readable medium that include accessing a frame buffer of a GPU, analyzing, in the frame buffer of the GPU, a first frame of displayed data from a source, the source being an image or video-based feed including an object, identifying the object in the first frame from the source, the object associated with a first value of a parameter, analyzing, in the frame buffer of the GPU, a second frame of the displayed data from the source, identifying the object in the second frame from the source, the object associated with a second value of the parameter, determining an entropy of the object based on a difference between the first parameter value and the second parameter value, generating a random number based on the entropy of the object, and generating an encryption for encrypting data based on the generated random number.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application is a continuation of U.S. application Ser. No.17/963,854, filed Oct. 11, 2022, which is a continuation of U.S.application Ser. No. 17/675,924, filed Feb. 18, 2022, which claimspriority to U.S. Provisional Application No. 63/272,790, filed Oct. 28,2021, and U.S. Provisional Application No. 63/182,391, filed Apr. 30,2021, the entire content of each of which is incorporated by referenceherein in its entirety for all purposes.

FIELD OF THE DISCLOSURE

The present disclosure relates to secure random number generation forencrypting data.

DESCRIPTION OF THE RELATED ART

Most of random functions are digital. When the need arises to derive arandom number for any technical use, an all-digital routine, all-digitalalgorithm, and all-digital formula can be used to generate the randomnumber, and digital systems, by nature, are inherently non-random, whichimpedes their ability to generate truly random numbers. As a result, ifattackers were to gather a sufficient amount of randomly generatednumbers, they would be able to potentially reverse-engineer the routine,the algorithm, or the formula to allow them to significantly reduce therange of possible numbers they would need to guess to find the generatednumbers, and modern commodity computing power allows guessing, or“brute-forcing,” billions of attempts in a relatively short timeframe,or start to learn how to mimic the routine, the algorithm, or theformula which were used to generate the random numbers. In addition, ifthe random numbers are a sub-component of security architecture or asecurity pattern, the attackers may unravel the random function.

The foregoing description is for the purpose of generally presenting thecontext of the disclosure. Work of the inventors, to the extent it isdescribed in this background section, as well as aspects of thedescription which may not otherwise qualify as prior art at the time offiling, are neither expressly or impliedly admitted as prior art againstthe present disclosure.

SUMMARY

The present disclosure relates to an apparatus, including: processingcircuitry, including a graphics processing unit (GPU), configured toaccess a frame buffer of the GPU, analyze, in the frame buffer of theGPU, a first frame representing a first section of a stream of displayeddata from a source, the first frame being displayed by the device, thesource being an image or video-based feed including an object, based onthe analyzed first frame, identify the object in the first frame fromthe source, the object associated with a parameter having a first valuecorresponding to the first frame of the source, analyze, in the framebuffer of the GPU, a second frame representing a second section of thestream of the displayed data from the source that is being displayed bythe device, based on the analyzed second frame, identify the object inthe second frame from the source, the object associated with a secondvalue of the parameter corresponding to the second frame, determine anentropy of the object based on a difference between the first value ofthe parameter of the object corresponding to the first frame of thesource and the second value of the parameter corresponding to the secondframe of the source, the entropy corresponding to a degree of randomnessof the object, generate a random number based on the entropy of theobject, and generate an encryption for encrypting data based on thegenerated random number.

The present disclosure also relates to a method, including: accessing aframe buffer of a graphics processing unit (GPU); analyzing, in theframe buffer of the GPU, a first frame representing a first section of astream of displayed data from a source, the first frame being displayedby a device, the source being an image or video-based feed including anobject; based on the analyzed first frame, identifying the object in thefirst frame from the source, the object associated with a parameterhaving a first value corresponding to the first frame of the source;analyzing, in the frame buffer of the GPU, a second frame representing asecond section of the stream of the displayed data from the source thatis being displayed by the device; based on the analyzed second frame,identifying the object in the second frame from the source, the objectassociated with a second value of the parameter corresponding to thesecond frame; determining an entropy of the object based on a differencebetween the first value of the parameter of the object corresponding tothe first frame of the source and the second value of the parametercorresponding to the second frame of the source, the entropycorresponding to a degree of randomness of the object; generating arandom number based on the entropy of the object; and generating anencryption for encrypting data based on the generated random number.

The present disclosure also relates to a non-transitorycomputer-readable storage medium for storing computer-readableinstructions that, when executed by a computer, cause the computer toperform a method, the method including: accessing a frame buffer of agraphics processing unit (GPU); analyzing, in the frame buffer of theGPU, a first frame representing a first section of a stream of displayeddata from a source, the first frame being displayed by a device, thesource being an image or video-based feed including an object; based onthe analyzed first frame, identifying the object in the first frame fromthe source, the object associated with a parameter having a first valuecorresponding to the first frame of the source; analyzing, in the framebuffer of the GPU, a second frame representing a second section of thestream of the displayed data from the source that is being displayed bythe device; based on the analyzed second frame, identifying the objectin the second frame from the source, the object associated with a secondvalue of the parameter corresponding to the second frame; determining anentropy of the object based on a difference between the first value ofthe parameter of the object corresponding to the first frame of thesource and the second value of the parameter corresponding to the secondframe of the source, the entropy corresponding to a degree of randomnessof the object; generating a random number based on the entropy of theobject; and generating an encryption for encrypting data based on thegenerated random number.

The foregoing paragraphs have been provided by way of generalintroduction and are not intended to limit the scope of the followingclaims. The described embodiments, together with further advantages,will be best understood by reference to the following detaileddescription taken in conjunction with the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

A more complete appreciation of the disclosure and many of the attendantadvantages thereof will be readily obtained as the same becomes betterunderstood by reference to the following detailed description whenconsidered in connection with the accompanying drawings, wherein:

FIG. 1 is a schematic view of user devices communicatively connected toa server, according to an exemplary embodiment of the presentdisclosure.

FIG. 2A is a flow chart for a method of generating a reference patch andembedding the reference patch into displayed data, according to anexemplary embodiment of the present disclosure.

FIG. 2B is a flow chart of a sub-method of generating the referencepatch, according to an exemplary embodiment of the present disclosure.

FIG. 2C is a flow chart of a sub-method of associating the surface areawith digital content, according to an exemplary embodiment of thepresent disclosure.

FIG. 2D is a flow chart of a sub-method of integrating the referencepatch into the displayed data, according to an exemplary embodiment ofthe present disclosure.

FIG. 3A is a flow chart for a method of inspecting the reference patch,according to an exemplary embodiment of the present disclosure.

FIG. 3B is a flow chart of a sub-method of identifying the referencepatch with unique identifiers corresponding to the surface area from thestream of data, according to an exemplary embodiment of the presentdisclosure.

FIG. 3C is a flow chart of a sub-method of associating the uniqueidentifiers with digital content, according to an exemplary embodimentof the present disclosure.

FIG. 4A is an illustration of a viewable area of a display, according toan exemplary embodiment of the present disclosure.

FIG. 4B is an illustration of a reference patch within a viewable areaof the display, according to an exemplary embodiment of the presentdisclosure.

FIG. 4C is an illustration of secondary digital content within aviewable area of the display, according to an exemplary embodiment ofthe present disclosure.

FIG. 5A illustrates an example of the first device generating randomnumber(s) based on an entropy value of one or more objects in one ormore images or videos using computer vision techniques, according to anembodiment of the present disclosure.

FIG. 5B is a schematic of the various layers and the objects receivedfrom the respective sources, according to an embodiment of the presentdisclosure.

FIG. 5C is a schematic of the analyzed object motion paths converted toentropy, according to an embodiment of the present disclosure.

FIG. 6 is a flowchart outlining a method, according to an embodiment ofthe present disclosure

FIG. 7 is a block diagram illustrating an exemplary electronic userdevice, according to certain embodiments of the present disclosure.

FIG. 8 is a schematic of a hardware system for performing a method,according to an exemplary embodiment of the present disclosure.

FIG. 9 is a schematic of a hardware configuration of a device forperforming a method, according to an exemplary embodiment of the presentdisclosure.

FIG. 10 is an example of Transparent Computing.

DETAILED DESCRIPTION

The terms “a” or “an”, as used herein, are defined as one or more thanone. The term “plurality”, as used herein, is defined as two or morethan two. The term “another”, as used herein, is defined as at least asecond or more. The terms “including” and/or “having”, as used herein,are defined as comprising (i.e., open language). Reference throughoutthis document to “one embodiment”, “certain embodiments”, “anembodiment”, “an implementation”, “an example” or similar terms meansthat a particular feature, structure, or characteristic described inconnection with the embodiment is included in at least one embodiment ofthe present disclosure. Thus, the appearances of such phrases or invarious places throughout this specification are not necessarily allreferring to the same embodiment. Furthermore, the particular features,structures, or characteristics may be combined in any suitable manner inone or more embodiments without limitation.

A method of random number generation in which attackers are not able tounravel the random function can provide stronger encryption whensecuring encoded data. Furthermore, given the inherent difficulty forprogrammed, digital systems to produce truly random data, an analogsource of random data is needed. Described herein, a method fordetecting digital objects during an augmented visual experience can beused to detect objects from various sources or inputs to provide theanalog source of random data for the encryption.

The present disclosure provides methods for generating augmented visualexperiences that are informative and interactive. According to anembodiment, the present disclosure relates to augmentation of a digitaluser experience. The augmentation may include an overlaying of digitalobjects onto a viewable display area of a display of an electronicdevice. The electronic device may be a mobile device such as asmartphone, tablet, and the like, a desktop computer, or any otherelectronic device that displays information. The digital objects mayinclude text, images, videos, and other graphical elements, amongothers. The digital objects may be interactive. The digital objects maybe associated with third-party software vendors.

In order to realize the augmentation of a digital user experience, areference patch, that is a region of interest acting as an anchor, canbe used. In one embodiment, the reference patch or other visuallydetectable element may serve to indicate a position at which digitalcontent is to be placed onto a display. In some embodiments and asdescribed herein, the reference patch may include encoded informationthat may be used to retrieve digital content and place that digitalcontent into a desired location or locations in displayed data. Thereference patch can be embedded within displayed data (such as, but notlimited to, an image, a video, a document, a webpage, or any otherapplication that may be displayed by an electronic device). Thereference patch can include unique identifying data, a marker, orencoding corresponding to predetermined digital content. The referencepatch can indicate to the electronic device the particular content thatis to be displayed, the position at which the content is to be placed,and the size with which the content is to be displayed. Accordingly,when a portion of the displayed data including the reference patch isvisible in a current frame of displayed data, the correspondingaugmentation can be overlaid on the current frame of the displayed datawherein the augmentation includes secondary digital content (i.e.,content that is secondary to (or comes after) the primary displayeddata), herein referred to as “digital content,” and/or digital objects.For example, an augmentation can include additional images to bedisplayed with the current frame of displayed data for a seamless visualexperience.

The above-described augmentations are particularly relevant toenvironments where the underlying content is static. Static content mayinclude textual documents or slide decks. Often, the static content isstored locally in the electronic device. Due to its nature, the staticcontent is not capable of being dynamically adjusted according tocomplex user interactions, in real-time, during a user experience. Sucha digital user experience is cumbersome and inefficient. Thus, aheightened, augmented user experience is desired to provide increasedconvenience, engagement, and agility. The augmentations described hereinreduce cumbrousness by providing a visual representation/aid ofretrieved external digital content, and provide improved engagement ofthe user, agility of navigation through the displayed data, and overallperformance of the user device.

Described herein is a device and method to incorporate a reference patchwith encoded identifier attributes, where the reference patch serves asa conduit for delivering content into the displayed data.

Referring now to the figures, FIG. 1 is a schematic view of anelectronic device, such as a client/user device (a first device 701)communicatively connected, via a network 851, to a second electronicdevice, such as a server (a second device 850), and a generating device1001, according to an embodiment of the present disclosure. Further, inan embodiment, additional client/user devices can be communicativelyconnected to both the first device 701 and the second device 850. Asecond client/user device (a third device 702) can be communicativelyconnected to the first device 701 and the second device 850. As shown, aplurality of the client/user devices can be communicatively connectedto, for example, an Nth user device 70 n.

An application may be installed or accessible on the first device 701for executing the methods described herein. The application may also beintegrated into the operating system (OS) of the first device 701. Thefirst device 701 can be any electronic device such as, but not limitedto, a personal computer, a tablet pc, a smart-phone, a smart-watch, anintegrated AR/VR (Augmented Reality/Virtual Reality) headwear with thenecessary computing and computer vision components installed (e.g., acentral processing unit (CPU), a graphics processing unit (GPU),integrated graphics on the CPU, etc.), a smart-television, aninteractive screen, a smart projector or a projected platform, an IoT(Internet of things) device or the like.

As illustrated in FIG. 1 , the first device 701 includes a CPU, a GPU, amain memory, and a frame buffer, among other components (discussed inmore detail in FIGS. 7-9 ). In an embodiment, the first device 701 cancall graphics that are displayed on a display. The graphics of the firstdevice 701 can be processed by the GPU and rendered in scenes stored onthe frame buffer that is coupled to the display. In an embodiment, thefirst device 701 can run software applications or programs that aredisplayed on a display. In order for the software applications to beexecuted by the CPU, they can be loaded into the main memory, which canbe faster than a secondary storage, such as a hard disk drive or a solidstate drive, in terms of access time. The main memory can be, forexample, random access memory (RAM) and is physical memory that is theprimary internal memory for the first device 701. The CPU can have anassociated CPU memory and the GPU can have an associated video or GPUmemory. The frame buffer may be an allocated area of the video memory.The GPU can display the displayed data pertaining to the softwareapplications. It can be understood that the CPU may have multiple coresor may itself be one of multiple processing cores in the first device701. The CPU can execute commands in a CPU programming language such asC++. The GPU can execute commands in a GPU programming language such asHLSL. The GPU may also include multiple cores that are specialized forgraphic processing tasks. Although the above description was discussedwith respect to the first device 701, it is to be understood that thesame description applies to the other devices (701, 702, 70 n, and 1001)of FIG. 1 . Although not illustrated in FIG. 1 , the second device 850can also include a CPU, GPU, main memory, and frame buffer.

FIG. 2A is a flow chart for a method 200 of generating a reference patchand embedding the reference patch into displayed data, according to anembodiment of the present disclosure. The present disclosure describesgeneration of the reference patch and embedding of this patch into thedisplayed data content in order to integrate additional content on thefirst device 701. In an embodiment, the first device 701 can incorporatedigital content into what is already being displayed (displayed data)for a more immersive experience.

In this regard, the first device 701 can generate the reference patch instep 205. The reference patch can be an object having an area and shapethat is embedded in the displayed data at a predetermined location inthe displayed data. For example, the reference patch can be a squareoverlayed and disposed in a corner of a digital document (an example ofdisplayed data), wherein the reference patch can be fixed to apredetermined page for a multi-page (or multi-slide) digital document.The reference patch can thus also represent a region of interest in thedigital document. The reference patch can be an object that, when not ina field of view of the user, is inactive. The reference patch can, uponentering the field of view of the user, become active. For example, thereference patch can become active when detected by the first device 701in the displayed data. When active, the reference patch can retrievedigital content and augment the displayed data by incorporating theretrieved digital content into the displayed data. Alternatively, thereference patch can become active when being initially located withinthe frame of the screen outputting the displayed data. For example, evenif another window or popup is placed over top of the reference patch,the reference patch may continue to be active so long as the referencepatch remains in the same location after detection and the windowincluding the document incorporating the reference patch is notminimized or closed. As will be described further below, the referencepatch can have a predetermined design that can be read by the firstdevice 701, leading to the retrieval and displaying of the digitalcontent.

In an embodiment, the first device 701 can use a geometrical shape forthe reference patch for placement into any displayed data usingapplications executed in the first device 701. The reference patch cantake any shape such as a circle, square, rectangle or any arbitraryshape. In step 210, the reference patch can also have predeterminedareas within its shape for including predetermined data. Thepredetermined data can be, for example, unique identifiers thatcorrespond to a surface area of the displayed data. The uniqueidentifiers can be, for example, a marker. As will be described below,the marker can take the form of patterns, shapes, pixel arrangements,pixel luma, and pixel chroma, among others. The surface area, by way ofthe unique identifiers, can be associated with predetermined digitalcontent that is recalled and displayed at the corresponding surface areain the displayed data. The unique identifier can include encoded datathat identifies the digital content, a location address of the digitalcontent at the second device 850 (see description below), a screenposition within the surface area at which the digital content isinsertable in the displayed data, and a size of the digital content wheninserted in the displayed data (adjustable before being displayed).

That is, in an embodiment, the surface area (or an available area inwhich digital content is insertable/to be inserted) of the displayeddata can be portion(s) of the displayed data that do not include objectsthat might obscure the reference patch or the digital content displayedat the corresponding surface area in the displayed data. For example,the first device 701 can use computer vision (described below) to detectthe objects. For example, the first device 701 can inspect an array todetermine locations of the objects. For example, a slide in a slide deckcan include text, pictures, logos, and other media, and the surface areacan be the blank space or spaces around the aforementioned objects.Thus, the digital content can be displayed somewhere in the blankspaces. In an embodiment, the surface area of the displayed data caninclude portions of the displayed data that already include objects andthe digital content can be displayed at the same location as theobjects. For example, a slide in a slide deck can include a picture of auser, and the reference patch can be the area representing a face of theuser and the digital content can be displayed at the same location as abody of the user. For example, a slide in a slide deck can include animage of a vehicle and the reference patch can be disposed in a blankspace of the displayed data, while the digital content retrieved (e.g.,a new car paint color and new rims) can be displayed over the image ofthe vehicle. In other words, the digital content may be placed in ablank area of the displayed data and/or in an area that is not blank(i.e., an area that includes text, image(s), video(s), etc.).

In step 215, the first device 701 can embed the reference patch into thedisplayed data, such as a word processing document file (i.e., DOC/DOCX)provided by e.g., Microsoft® Word, in a Portable Document Format (PDF)file such as the ones used by Adobe Acrobat®, in a Microsoft® PowerPointpresentation (PPT/PPTX), or in a video sequence file such as MPEG, MOV,AVI or the like. These file formats are illustrative of some file typeswhich a user may be familiar with; however, applications included in thefirst device 701 are not limited to these types and other applicationsand their associated file types are possible.

The reference patch (or similar element) can be embedded into anydisplayed data, where the displayed data may be generated by anapplication running on or being executed by the first device 701. Thereference patch can encompass the whole area designated by the displayeddata, or just a portion of the area designated by the displayed data.The method of generating the reference patch and embedding the referencepatch into the displayed data has been described as being performed bythe first device 701, however, the second device 850 can instead performthe same functions. In order to be detected in the displayed data on thefirst device 701, the reference patch may only be simply displayed as animage on the screen. The reference patch may also simply be a rasterimage or in the background of an image. The reference patch is also ableto be read even when the image containing the reference patch is lowresolution. Because the reference patch is encoded in a hardy andenduring manner such that even if a portion of the reference patch iscorrupted or undecipherable, the reference patch can still be activatedand used.

In an embodiment, the reference patch can be embedded inside of a bodyof an email correspondence. The user can use any electronic mailapplication such as Microsoft Outlook®, Gmail®, Yahoo®, etcetera. As theapplication is running on the first device 701, it allows the user tointeract with other applications. In an embodiment, the reference patchcan be embedded on a video streaming or two-way communication interfacesuch as a Skype® video call or a Zoom® video call, among others. In anembodiment, the reference patch can be embedded in displayed data formulti-party communication on a live streaming interface such as Twitch®.

One way in which the first device 701 may embed the reference patch intothe displayed data is by arranging the generated reference patch in thedisplayed data such as in a desired document or other media. Thereference patch may include a facade of the digital content whichbecomes an integrated part of the displayed data. The facade can act asa visual preview to inform the user of the digital content linked to thereference patch. The facade can include, for example, a screenshot of avideo to be played, a logo, an animation, or an image thumbnail, amongothers. The facade can be a design overlay. The design overlay can be apicture that represents the underlying digital content superimposed overthe reference patch. In an embodiment, the facade can indicate thecontent that is represented by the reference patch. The facade can becontained within the shape of the reference patch or have a dynamicsize. For example, attention of the user can be brought to the facade byadjusting the size of the facade when the reference patch is displayedon the display. The adjustment of the size of the facade can also bedynamic, wherein the facade can enlarge and shrink multiple times. Bythe same token, a position and rotation of the facade can also beadjusted to produce a shaking or spinning effect, for instance.

Unlike traditional means of sending displayed data, the first device 701may not send the whole digital content with a header file (metadata) anda payload (data). Instead, the reference patch that may include a facadeof the underlying digital content is placed within the displayed data.If a facade is used, it indicates to the first device 701 that thesurface area can have digital content that can be accessed withselection (clicking with a mouse, touchpad, eye-gaze, eye-blinks, or viavoice-command) of the facade. The digital content can also be accessedor activated automatically, e.g., when the user has the reference patchdisplayed on the display of the first device 701. Other symbolic meansof visualization can be employed to indicate to the user that thesurface area is likely to include information for obtaining digitalcontent. For example, a highlighting effect can be applied along aperimeter of the reference patch in a pulsating pattern of highlightingintensity to bring attention to the presence of the reference patch. Forexample, a series of spaced dashes surrounding the reference patch andoriented perpendicular to the perimeter of the reference patch canappear and disappear to provide a flashing effect. Other means can beemployed to indicate to the user that the surface area is likely toinclude information for obtaining digital content, such as an audio cue.

The first device 701 employs further processes before embedding thereference patch into the displayed data. These processes and schemas arefurther discussed in FIG. 2B.

FIG. 2B is a flow chart of a sub-method of generating the referencepatch, according to an embodiment of the present disclosure. The firstdevice 701 can associate the digital content with the surface areacorresponding to the reference patch (e.g., via the unique identifiersincluded therein) generated by the first device 701. In an embodiment,the surface area may encompass the whole of the displayed data or aportion of it.

The reference patch, which includes the unique identifiers correspondingto the surface area associated with the digital content, is thenembedded into the displayed data by the first device 701. In some usecases, the displayed data including the reference patch can be sent ortransmitted to a second user having the third device 702 including thesame application, which then allows the second user to accessinformation within the surface area and obtain the digital content andhave it viewable on the third device 702. That is, the third device 702can have the same displayed data overlaid with the augmenting digitalcontent on the surface area of the display of the third device 702 inthe location or locations defined by the reference patch.

In FIG. 2B, the generating device 1001 uses additional processes toeffectuate generation of the reference patch which is obtained andembedded by the first device 701. In an embodiment, the generatingdevice 1001 encodes the reference patch with the unique identifierscorresponding to the surface area in step 205 a. The generating device1001 can mark areas of the reference patch in step 205 b to form themarker that, either separately or in combination, define or may be usedto access the unique identifiers. The marker can take the form ofpatterns, shapes, pixel arrangements, or the like. In an example, themarker can have a shape that corresponds to the shape of the surfacearea. In an example, the marker can have a size that corresponds to thesize of the surface area. In an example, the marker can have a perimeterthat corresponds to the perimeter of the surface area. The marker canuse any feasible schema to provide identifying information thatcorresponds to the surface area within parts of the displayed data. Inan embodiment, the marker can incorporate hidden watermarks that areonly detectable by the first device 701 and the third device 702, whichhave detection functionality implemented therein, for example having theapplication installed or the functionality built into the operatingsystem.

The marker can incorporate patterns which can then be extracted by thefirst device 701. In an example, the first device 701 can perform theembedding, then send the digital content having the embedded referencepatch to the third device 702. The encoding is performed by thegenerating device 1001 and may use any variety of encoding technologiessuch as the ARUCO algorithm to encode the reference patch by marking thereference patch with the marker. The first device 701 may also be usedas the generating device 1001.

In an embodiment, the marker can be comprised of a set of points,equidistant from each other and/or some angle apart from a referencepoint, such as the center of the reference patch or represent some otherfiducial points. That is, the fiducial points corresponding to themarker can provide a set of fixed coordinates or landmarks within thedigital content with which the surface area can be mapped relative tothe fiducial points. In an embodiment, the marker can be comprised of aset of unique shapes, wherein predetermined combinations of the uniqueshapes can correspond to a target surface area (or available area, orareas) for displaying the displayed data. The predetermined combinationsof the unique shapes can also correspond to predetermined digitalcontent for displaying in the surface area. The predeterminedcombinations of the unique shapes can also correspond to/indicate aposition/location where the digital content should be displayed at thesurface area relative to a portion of the surface area. A combination ofthe set of points and unique identifiers can be used as well.

For example, the unique identifiers can be unique shapes that correlateto predetermined digital content as well as indicating where the digitalcontent should be overlayed on the display (the screen position)relative to a set of points marked on the reference patch. The uniqueidentifiers can also indicate a size of the digital content to beoverlayed on the display, which can be adjustable based on the size ofthe surface area (also adjustable) and/or the size of the display of thefirst device 701. The unique identifiers can be relatively invisible orundetectable to the user, but readable by the first device 701 and coverpredetermined areas of the reference patch. The unique identifiers, andby extension, the marker, can have an appearance that is marginallydifferent from an appearance of the area of the reference patch. Forexample, the area of the reference patch can appear white to the userand the unique identifiers can also appear white to the user but mayactually have a slightly darker pixel color that can be detected andinterpreted by a device, such as the first device 701. For instance, theappearance of the unique identifiers can be 0.75% darker than the whitecolor of the area of the reference patch. Such a small difference can beidentified and discerned by the first device 701 while beingsubstantially imperceptible to the user.

In an embodiment, the area of the reference patch can be divided intopredetermined shapes, for instance a set of squares, and within eachsquare, the marker (such as a “letter”) can be included. For example,there can be 16 squares. Furthermore, subsets of the set of squares canbe designated to represent varying information, such as a timestampcorresponding to 8 of the squares, a domain corresponding to 5 of thesquares, a version corresponding to 1 of the squares, and additionalinformation corresponding to a remainder of the squares. Anidentification based on the set of squares can be, for example, an18-character (or “letter”) hexadecimal. The set of squares can furtherinclude additional subsets for a randomization factor, which can be usedfor calculating a sha256 hash prior to encoding the reference patch withthe hash. Together, the set of squares having the marker includedtherein can comprise the unique identifiers.

Moreover, the generating device 1001 can also employ chroma subsamplingto mark attributes represented by a particular pattern. In anembodiment, the generating device 1001 can mark parts of the referencepatch with predetermined patterns of pixel luma and chroma manipulationthat represent a shape, a size, or a position of the surface area fordisplaying the digital content. Moreover, the generating device 1001 canmark a perimeter of the reference patch with a predetermined edgingpattern of pixel luma and chroma manipulation that represents aperimeter of the surface area for displaying the digital content.

The generating device 1001 can further link the surface area with uniqueidentifiers in step 205 c. The unique identifiers can be hashed values(such as those described above) that are generated by the generatingdevice 1001 when the reference patch is generated (such as the onehaving the area of the reference patch divided into the subset ofsquares).

FIG. 2C is a flow chart of a sub-method of associating the surface areawith digital content, according to an embodiment of the presentdisclosure. In FIG. 2C, the generating device 1001 uses additionalprocesses to associate the surface area with digital content. In anembodiment, the generating device 1001 can associate the uniqueidentifiers corresponding to the surface area with metadata. In step 210a, the unique identifiers can be associated with metadata embodyinginformation about the storage and location of the digital content.Moreover, in step 210 b, the generating device 1001 can associate theunique identifier of the surface area with metadata which embodiesinformation about the format and rendering information used for thedigital content. In step 210 c, the generating device 1001 can associatethe unique identifiers of the surface area with metadata which embodiesaccess control information of the digital content.

In an embodiment, the storage of the digital content can be on a remoteserver, such as the second device 850, and the location of the digitalcontent can be the location address of the memory upon which it isstored at the remote server. The storage and location of the digitalcontent are thus linked with the metadata that can point to where thedigital content can later be obtained from. The digital content is notembedded into the displayed data. In an embodiment, the format andrendering information about the digital content is embodied in themetadata and associated with the unique identifiers. This information ishelpful when the first device 701 or the third device 702 are on thereceiving end of the transmitted displayed data and need to properlyretrieve and process the digital content.

Moreover, in an embodiment, the access control of the digital contentcan also be encompassed in the metadata and associated with the uniqueidentifiers corresponding to the surface area. The access control can beinformation defining whether the digital content can be accessed bycertain individuals or within a certain geographical location. Theaccess control information can define restrictions such as those placedupon time and date as to when and how long the digital content can beaccessed. The access control information can define the type of displayreserved for access by the first device 701. For example, a user maywish to restrict access to the digital content to certain types ofdevices, such as smartphone or tablets. Thus, the metadata defining adisplay requirement would encompass such an access control parameter.

FIG. 2D is a flow chart of a sub-method of integrating the referencepatch into the displayed data, according to an embodiment of the presentdisclosure. In FIG. 2D, the generating device 1001 uses additionalprocesses to effectuate integration of the reference patch into thedisplayed data. In an embodiment, the first device 701 can temporarilytransfer or store the reference patch in a storage of the first device701 in step 215 a. The storage can be accessed by the first device 701for embedding the reference patch into the displayed data at any time.The first device 701 can extract the reference patch from the storagefor embedding purposes in step 215 b. The first device 701 can alsoarrange the reference patch at a predetermined location and with apredetermined reference patch size in step 215 c. The first device 701can further embed the reference patch such that a document, for example,having the reference patch embedded therein can be sent to a recipient,for example the second user using the third device 702, where he/she canaccess the document using the application on the third device 702 asfurther described below. Again, the features of the generating device1001 can be performed by the first device 701.

The displayed data can be output from a streaming application or acommunication application with a data stream having the reference patchembedded therein. The actual digital content may not be sent along withthe underlying displayed data or data stream, but only the uniqueidentifier and/or a facade of the digital content is sent. The uniqueidentifier and/or the underlying metadata can be stored in a cloud-baseddatabase such as MySQL which can point to the second device 850 or acloud-based file hosting platform that ultimately houses the digitalcontent. No limitation is to be taken with the order of the operationdiscussed herein; such that the sub-methods performed by the firstdevice 701 can be carried out synchronous to one another, asynchronous,dependently or independently of one another, or in any combination.These stages can also be carried out in serial or in parallel fashion.

FIG. 3A is a flow chart for a method 300 of identifying the referencepatch included in the displayed data and overlaying the digital contentinto displayed data, according to an embodiment of the presentdisclosure. In an embodiment, in step 305, the first device 701 caninspect the stream of data being outputted by the first device's 701video or graphics card and onto the display of the first device 701.That is, the first device 701 can access a frame buffer of the GPU andanalyze, frame by frame, in the frame buffer, the outputted stream ofdata which can include the displayed data. In an embodiment, a framerepresents a section of the stream of the displayed data that is beingdisplayed by the first device 701. In that regard, the first device 701can inspect the outputted stream of data. The first device 701 canachieve this by intercepting and capturing data produced from the firstdevice 701's video card or GPU that is communicated to the first device701's display.

In an embodiment, in step 310, the first device 701 can processattributes of each pixel included in a single frame and detect groups ofpixels within that frame, which may have a known predetermined patternof pixel luma and chroma manipulation, in order to find the referencepatch.

The inspected frame by frame stream of data is also used by the firstdevice 701 to identify the reference patch which includes the uniqueidentifiers therein. In an embodiment, the first device 701 employspattern recognition algorithms to detect and identify the referencepatch, the perimeter of the reference patch, and/or the area of thereference patch. In an embodiment, the first device 701 detects andidentifies the reference patch via the marker itself. A variety ofpattern recognition algorithms can be used, such as Artificial NeuralNetworks (ANN), Generative Adversarial Networks (GAN), thresholding, SVM(Support Vector Machines) or any classification and pattern recognitionalgorithm available conducive to computer vision. Computer visiontechniques may be artificial intelligence techniques that traincomputers to interpret and understand a visual world. In an example, thecomputer vision techniques may be an image recognition task, a semanticsegmentation task, and the like. In a non-limiting example, theprocessor-based computer vision operation can include sequences offiltering operations, with each sequential filtering stage acting uponthe output of the previous filtering stage. For instance, when theprocessor (processing circuitry) is/includes a GPU, these filteringoperations are carried out by fragment programs.

In another example, the computer vision techniques may be aprocessor-based computer vision technique. In an embodiment, the firstdevice 701 can look for predetermined or repeatable patterns within theframe which indicates the presence of the reference patch. In anembodiment, the first device 701 can identify the reference patch basedon a confidence level, the confidence level being high when thepredetermined pattern of pixel luma and chroma manipulation and thepredetermined edging pattern of pixel luma and chroma manipulation aredetected in the reference patch. The confidence level can be lower whenone or neither of the predetermined patterns is/are detected.

According to an embodiment, in the event that an input to the operationis an image, the input images can be initialized as textures and thenmapped onto quadrilaterals. By displaying these quadrilaterals inappropriately sized windows, a one-to-one correspondence of image pixelsto output fragments can be ensured. Similarly, when the input to theoperation is an encoded image, a decoding process may be integrated intothe processing steps described above. A complete computer visionalgorithm can be created by implementing sequences of these filteringoperations. After the texture has been filtered by the fragment program,the resulting image is placed into texture memory, either by usingrender-to-texture extensions or by copying the frame buffer into texturememory. In this way, the output image becomes the input texture to thenext fragment program. This creates a pipeline that facilitates theentire computer vision algorithm. However, often a complete visionalgorithm may require operations beyond filtering. For example,summations are common operations. Furthermore, more-generalizedcalculations, such as feature tracking, can also be mapped effectivelyonto graphics hardware.

In an embodiment, the reference patch can be identified by use of edgedetection methods. In particular, edge detection can be used for theperimeter of the reference patch having a predetermined pattern (thepredetermined edging pattern). In an example, the edge detection methodmay be a Canny edge detector. The Canny edge detector may run on theGPU. In one instance, the Canny edge detector can be implemented as aseries of fragment programs, each performing a step of the algorithm.

In an embodiment, the identified reference patch can be tracked fromframe to frame using feature vectors. Calculating feature vectors atdetected feature points is an operation in computer vision. A feature inan image is a local area around a point with some higher-than-averageamount of uniqueness. This makes the point easier to recognize insubsequent frames of video. The uniqueness of the point is characterizedby computing a feature vector for each feature point. Feature vectorscan be used to recognize the same point in different images and can beextended to more generalized object recognition techniques.

Feature detection can be achieved using methods similar to the Cannyedge detector that instead search for corners rather than lines. If thefeature points are being detected using sequences of filtering, the GPUcan perform the filtering and read back to the CPU a buffer that flagswhich pixels are feature points. The CPU can then quickly scan thebuffer to locate each of the feature points, creating a list of imagelocations at which feature vectors on the GPU will be calculated.

In step 315, the first device 701 can decode the encoded data of theunique identifiers from the area of the reference patch, wherein theunique identifiers correspond to the surface area. The uniqueidentifiers can be hashed values that could have been generatedbeforehand by the first device 701.

In step 320, the first device 701 can use the unique identifiers to linkthe surface area with the digital content using metadata and retrievethe digital content based on the unique identifiers.

In step 325, the first device 701 can overlay the digital content ontothe surface area of the displayed data based on the unique identifiers.

Again, the method of identifying the reference patch included in thedisplayed data and augmenting the displayed data is described asperformed by the first device 701, however, the second device 850 caninstead perform the same functions.

In an embodiment, the first device 701 identifies the surface areacorresponding to the reference patch by employing further processes toprocess the frames. To this end, FIG. 3B is a flow chart of a sub-methodof identifying the reference patch with the unique identifierscorresponding to the surface area from the stream of data, according toan embodiment of the present disclosure.

In step 310 a, the first device 701 can decode the encoded referencepatch from the frame. The encoded reference patch can include the markerthat makes up the unique identifiers within the reference patchincorporated previously. The reference patch can also include otheridentifying information. The marker can be disposed within the referencepatch, such as within the area of the reference patch or along aperimeter of the reference patch, or alternatively, outside of the areaof the reference patch.

Whatever schema is used to encode the marker in the reference patch isalso used in reverse operation to decode the underlying informationcontained within the reference patch. As stated above, in an embodiment,the encoded marker can be patterns generated and decoded using the ARUCOalgorithm or by other algorithms that encode data according to apredetermined approach.

In step 310 b, the first device 701 can also extract attributes of thesurface area from the reference patch. In an embodiment, the position,size, shape, and perimeter of the surface area are extracted, althoughother parameters can be extracted as well. Other parameters includeboundary lines, area, angle, depth of field, distance, ratio of pairs ofpoints, or the like. In an embodiment, where shape and perimeter aredesignated as the attributes, the first device 701 makes determinationsof size, shape, and perimeter and outputs that result. Specifically, thesize or shape of the surface area can be determined by evaluating apredetermined or repeatable pattern of pixel luma and chromamanipulation in the reference patch. The predetermined pattern can bemarked on, within the area, or outside of the area of the referencepatch. The predetermined pattern can correspond to the size or shape ofthe surface area. The predetermined pattern can correspond to the sizeor shape of the digital content. The perimeter of the surface area canalso be determined by evaluating a predetermined edging pattern of pixelluma and chroma manipulation. The predetermined edging pattern can bemarked on, within the area, or outside of the area of the referencepatch. That is, the predetermined edging pattern of the refence patchcan correspond to the perimeter of the surface area. The predeterminededging pattern of the refence patch can correspond to the perimeter ofthe digital content.

In step 310 c, the first device 701 can also calculate a position andsize of the surface area relative to the size and shape (dimensions) ofthe output signal from the display that is displaying the displayeddata. In an embodiment, the calculating of the size, relative to thesize and shape of the outputted signal from the display, includesdetermining the size of the surface area by inspecting a furthestmeasured distance between the edges of the surface area. Furthermore,the calculating of a location of the surface area, relative to the sizeand shape of the outputted signal from the display, includes determiningthe location of the surface area relative to the size and shape of thedisplayed data outputted through the display. This includes calculatingthe distance between the outer edges of the surface area and the inneredges of the displayed data being outputted by the display. Thedetermined size and location of the surface area can be outputted as aresult. Notably, prior to overlaying the digital content into thedisplayed data, the first device 701 can adjust, based on thepredetermined pattern and the predetermined edging pattern, the size andperimeter of the digital content for displaying in the display of thefirst device 701. For example, the size and perimeter of the digitalcontent for displaying in the display of the first device 701 can bescaled based on the size and perimeter of the surface area and/or thesize of the display.

The first device 701 can provide information regarding thecharacteristics of the output video signal, such that the digitalcontent that is later overlaid can correctly be displayed to account forvarious manipulations or transformations that may take place due tohardware constraints, user interaction, image degradation, orapplication intervention. Such manipulations and transformations may bethe relocation, resizing, and scaling of the reference patch and/or thesurface area, although the manipulations and transformations are notlimited to those enumerated herein. Such manipulations andtransformations may be the relocation, resizing, and scaling of thereference patch and/or the surface area, although the manipulations andtransformations are not limited to those enumerated herein.

In an embodiment, the reference patch itself can be used as thereference for which the digital content is displayed on the surfacearea. In one example, the location at which to display the digitalcontent in the surface area can be determined relative to the locationof the reference patch on the displayed data. In one example, the sizeof the surface area can be determined relative to the size of thereference patch on the displayed data. In an example employing acombination of the two properties of the reference patch, the referencepatch displayed in the displayed data on a smart phone having apredetermined size and a surface area can be scaled relative to thepredetermined size of the display of the smart phone. This can befurther adjusted when the reference patch in the same displayed data isdisplayed on a desktop monitor, such that the predetermined size of thereference patch in the displayed data displayed on the desktop monitoris larger and thus the size of the surface area can be scaled to belarger as well. Furthermore, the location of the surface area can bedetermined via a function of the predetermined size of the referencepatch. For example, the location at which to display the digital contentin the surface area can be disposed some multiple widths laterally awayfrom the location of the reference patch as well as some multipleheights longitudinally away from the location of the reference patch. Assuch, the predetermined size of the reference patch can be a function ofthe size of the display of the first device 701. For example, thepredetermined size of the reference patch can be a percentage of thewidth and height of the display, and thus the location and the size ofthe surface area are also a function of the width and height of thedisplay of the first device 701.

In an embodiment, the first device 701 can determine an alternativelocation at which to display the digital content based on behaviors ofthe user. For example, the first device 701 can compare the encoded datacorresponding to the location at which to display the digital content inthe surface area to training data describing movement and focus of theuser's eyes while viewing the displayed data. Upon determining thelocation at which to display the digital content in the surface area (asencoded in the reference patch) is not the same as the training data,the first device 701 can instead display the digital content at thelocation described by the training data as being where the user's eyesare focused in the displayed data at a particular time. For example, theuser's eyes may be predisposed to viewing a bottom-right of a slide in aslide deck. The first device 701 can decode the reference patch anddetermine the digital content is to be displayed in a bottom-left of theslide deck. The training data can indicate that, for example, the user'seyes only focus on the bottom-left of the slide 10% of the time, whileuser's eyes focus on the bottom-right of the slide 75% of the time.Thus, the first device 701 can then display the digital content in thebottom-right of the slide instead of the bottom-left. The training datacan also be based on more than one user, such as a test populationviewing a draft of the slide deck. For example, the training data can bebased on multiple presentations of the slide deck given to multipleaudiences, wherein eye tracking software determines the average locationof the audience's focus on each of the slides.

In an embodiment, the first device 701 employs other processes toassociate the unique identifiers with the digital content. To this end,FIG. 3C is a flow chart of a sub-method of associating the uniqueidentifiers with digital content, according to an embodiment of thepresent disclosure. In step 320 a, the first device 701 can send theunique identifiers to the second device 850 and the second device 850can retrieve metadata that describes the digital content, the digitalcontent being associated with the surface area through the uniqueidentifiers. This can be done by querying a remote location, such as adatabase or a repository, using the unique identifiers of the surfacearea as the query key. In an embodiment, the first device 701 sends theunique identifiers to the second device 850 and the second device 850associates the unique identifier of the reference patch to correspondingdigital content based on the metadata. The metadata associated with thesurface area's unique identifier can be transmitted to the first device701 with the augmentation content.

In step 320 b, the first device 701 can assemble the digital contentthat is associated with the surface area's unique identifier. Theassembly can entail loading the necessary assets for assembling thedigital content. In an embodiment, this can entail loading manipulationsoftware or drivers in order to enable the first device 701 to processthe digital content. Other assembling processes can be the loading ofrendering information in order to transform and manipulate an individualportion of the digital content. Furthermore, the loaded manipulationsoftware, drivers, or rendering information can be used to compile allthe individual portions of the entire digital content together. In anembodiment, this can include adapting the file formats of the digitalcontent, delaying the playback for the digital content, converting fromone format to another, scaling the resolution up or down, converting thecolor space, etc.

In step 320 c, the first device 701 can provide access controlparameters for the digital content. The access control parameters candictate whether the digital content is visible to some users, or to somegeographical locations, or to some types of displays and not others, aswell as the date and time or duration of time a user can access thedigital content or is allowed to access. In an embodiment, visibility ofthe digital content can be defined for an individual. For example, thedigital content can be a video that is appropriate for users over acertain age. In an embodiment, visibility of the digital content can bedefined for a geographic location. For example, the digital content canbe a video that is region-locked based on a location of the first device701. In an embodiment, visibility of the digital content can be definedfor a type of display displaying the displayed data. For example, thedigital content can be VR-based and will only display with a VR headset.In an embodiment, visibility of the digital content can be defined for apredetermined date and a predetermined time. For example, the digitalcontent can be a video that will only be made publicly available after apredetermined date and a predetermined time. In an embodiment,visibility of the digital content can be defined for a time period. Forexample, the digital content can be a video that is only available forviewing during a holiday. The first device 701 thus calculates theuser's access level based on those parameters and provides an outputresult as to the user's ability to access the digital content, i.e.,whether the digital content will be visible or invisible to the user.Note that the access control parameters can be global, for all thedisplayed data, or it can be localized per surface area and theunderlying digital content.

Referring again to FIG. 3A, in step 325, the first device 701 can carryon the processes of overlaying the surface area with the digital contentinto the displayed data in accordance with the surface area, theposition, and the size identified by the unique identifier. The firstdevice 701 can determine or adjust the size and location of theassembled digital content on the surface area relative to the size andshape of the displayed data being outputted by the display. Then, thefirst device 701 can render the associated digital content (or theassembled individual portions) over the surface area's shape andperimeter using the size and location information. Thus, the digitalcontent is superimposed on top of the surface area.

In yet another illustrative example of content augmentation, as will bedescribed with reference to FIG. 4A through FIG. 4C, the displayed datais a slide deck. The slide deck may be generated by a concierge-typeservice that seeks to connect a client with potential garden designers.As in FIG. 4A, the slide deck may be presented to the client within aviewable area 103 of a display 102 of the first device 701. Thepresently viewable content of the slide deck within the viewable area103 of the display 102 may be a current frame of display data 106.Traditionally, the slide deck may include information regarding eachpotential garden designer and may direct the client to third-partyapplications that allow the client to contact each designer. In otherwords, in order to connect with one or more of the potential gardendesigners, the client, traditionally, may need to exit the presentationand navigate to a separate internet web browser in order to learn moreabout the garden designers and connect with them. Such a digital userexperience can be also cumbersome.

With augmentation, however, the client need not leave the presentationin order to set up connections with the garden designers. For instance,as shown in FIG. 4B, a reference patch 104 can be positioned within theslide deck to be in the current frame 106 and viewable within theviewable area 103 of the display 102 at an appropriate moment. As shownin FIG. 4C, the reference patch 104 may correspond to digital content105 (i.e., one or more augmentations) and, when the reference patch 104is visible to/detectable by the first device 701, the digital content105 is retrieved and displayed by the first device 701 at thecorresponding surface area. The digital content 105 can include, asshown in FIG. 4C, interactive buttons, images, videos, windows, andicons, among others, that allow the client to interact with thedisplayed data and to, for instance, engage with the garden designerswithout leaving the presentation. In an example, the interactive digitalcontent 105 may allow for scheduling an appointment with a given gardendesigner while still within the slide deck.

As previously noted, the above-described augmentations are particularlyrelevant to environments where the underlying content is static. Staticcontent may include textual documents or slide decks. Often, the staticcontent is stored locally in the electronic device. Due to its nature,the static content is not capable of being dynamically adjustedaccording to complex user interactions, in real-time, during the userexperience.

Having obtained the digital content, the augmentation of the user visualexperience can be realized. This can include displaying, based on therendering instructions, the digital content relative to the referencepatch within a transparent layer of the display. The transparent layerof the display may be a topmost layer of the display so as to ‘overlay’the augmentation on the underlying displayed data.

The layer can refer to digital content displayed in a window of anelectronic device, such as the display of the user's device. Within awindow of the user's device, which can be a computer or smartphonedisplay, the device is typically configured to display a single layer ofcontent at a time. For example, in a traditional electronic device, if afull-screen Microsoft PowerPoint presentation (i.e. first layer) isbeing displayed on a device's window, that device cannot display afull-screen movie (i.e. second layer) without covering up the MicrosoftPowerPoint.

In comparison, as described herein, multiple layers of content can bemanaged, manipulated, and merged into a single computing experience. Inother words, multiple layers can be superimposed and displayedsimultaneously within the same display. In an embodiment, this isaccomplished by adjusting a transparency of pixels in one or more layersto achieve a semi-transparent effect. That way, any number ofsimultaneous media sources can be displayed in a window such that theblended input media can optically appear to exist either in front of orbehind other layers.

In an embodiment, a layer is a computer vision layer. The computervision layer may be a layer of the construction of explicit, meaningfuldescriptions of physical objects from images as determined by computervision. The content in the layer may be determined using image data froma first source (or input), e.g., a camera, or from a second source (orinput), e.g., a camera. The computer vision layer may be visible,semi-visible, or invisible to the users. The computer vision layer maybe further used, by the first device 101 or controlled by a server (e.g.the second device 150), to make useful decisions about real physicalobjects and scenes based on sensed images such as the one or moredigital images from, but not limited to, cameras, etc. In someembodiments, the layer may also be a video layer. In some embodiments,the layer may not include only one layer, it may include more than onelayer. Notably, even layers that are invisible to the user can be usedfor performing tasks and executing processes.

Described herein is a security mechanism for generatingcryptographically secure (pseudo) random numbers (CSPRNG). The presentdisclosure is also related to a mechanism to generate random number(s)based on an entropy value of one or more objects in digital contentssuch as one or more digital images or videos using computer visiontechniques.

In an embodiment, a first camera located in or around a state park cantake images or videos which include animals and vehicles, and a secondcamera located in or around a shopping mall can take images or videoswhich include humans, traffic signs, and vehicles. The images and thevideos from the first and the second cameras can be transmitted to adisplay of the first device 701, which can be a user's device. Thedisplay can be a display of a mobile device such as a smartphone,tablet, a display of a desktop computer, or another interactive display.The first device 701 can identify one or more objects in the images orthe videos displayed on the display of the first device 701. The one ormore objects can be, but are not limited to, animals, vehicles, trafficsigns, human, or the like. The first device 701 can calculate an entropyvalue for each of the one or more identified objects and determine thehighest entropy value among the calculated entropy values for generationof a random number, the entropy value corresponding to a degree ofrandomness of the object. That is, the entropy value of the one or moreobjects quantifies an average amount of information (e.g., bits) neededto represent an event drawn from a probability distribution for a randomvariable or metric. Objects with higher entropy values are moreuncertain/surprising/random than lower entropy values, and thereforeobjects with higher entropy values are harder to decode in acryptography-context. In one embodiment, the entropy value can bedetermined using min-entropy, wherein min-entropy measures a likelihoodof guessing a predetermined value on a first attempt.

In an embodiment, the random number generated based on the highestentropy values can be further used as seed information by an encryptionalgorithm (a cipher) to encode information such as the encoded data ofthe reference patch. For example, a scenario where user A sends an emailwith the encoded information encrypted by the random number based on thedetected object as an attached document to user B. Therefore, using therandom number generated based on the highest entropy values for theencoded information, attackers may not be able to unravel the randomfunction.

In an embodiment, the identification of the one or more objects can beperformed according to one of a variety of computer vision techniques.As previously described, computer vision techniques can be artificialintelligence techniques that train computers to interpret and understanda visual world. In an example, the computer vision techniques can be animage recognition task, a semantic segmentation task, and the like. Inan example, the computer vision techniques can be a processor-basedcomputer vision technique. In an example, the processor-based computervision operation can include sequences of filtering operations, witheach sequential filtering stage acting upon the output of the previousfiltering stage. For instance, when the processor is a GPU, thesefiltering operations can be carried out by fragment programs.

In an embodiment, random numbers can be used in various steps incryptography. For example, generating nonces, where numbers have beenonly used once, can require a random number. Initialization of vectorscan require a random number. Cryptographic keying materials can requirea random number. The security of basic cryptographic elements can dependon the underlying random number generator (RNG) that was used. An RNGsuitable for cryptographic usage is called a Cryptographically SecurePseudo-Random Number Generator (CSPRNG).

In an embodiment, the strength of cryptographic systems can depend onthe properties of the CSPRNGs. Depending on how the generatedpseudo-random data is applied, a CSPRNG should be unpredictable inadvance and not reliably reproduced after generation. In an embodiment,the strength of the CSPRNG can be directly proportional to the source ofentropy (source of randomness) used for seeding/re-seeding theencryption.

To this end, and referring to the Drawings, FIG. 5A illustrates anexample of the first device 701 generating random number(s) based on anentropy value of one or more objects in one or more images or videosusing computer vision techniques, according to an embodiment of thepresent disclosure. For example, as described earlier, a camera 106 inor around a state park can take/capture images or videos which includeanimals, vehicles, etc., and a camera 108 in or around a shopping mallcan take images or videos which include traffic signs, vehicles, etc.The first device 701 can be configured to receive input from one or moresources, such as the cameras 106 and 108. The images and the videos fromthe cameras 106 and 108 can be transmitted to the first device 701,temporarily stored in a memory of the first device 701, and displayed inone or more layers in the first device 701. For example, the memory ofthe first device 701 can include a memory buffer, wherein the memorybuffer is a region of the memory used to temporarily store data whilethe data is being moved from one location to another.

For example, the memory of the first device 701 can include the framebuffer of the GPU. That is, an individual layer of the first device 701can be configured to display input from one camera, two cameras, or morethan two cameras stored in the frame buffer. The input to be displayedin the layers, which can be, for example, the images and videos from thecameras 106 and 108, can be analyzed by the first device 701, andobjects from the input can be detected. The layer can be configured todisplay the input from the sources, but the displayed input may or maynot be visible to the user. Notably, the layer can have a transparency,which can be adjustable. For multiple layers, the transparency of eachlayer can be adjusted when displayed. The first device 701 can includeone or more layers with varying visibility, including some which can beinvisible to the user, while each layer of the one or more layers can beconfigured to display the input from the one or more sources, such asthe cameras 106 and 108.

In an embodiment, the first device 701 can identify one or more objectsin the images or the videos on the one or more layers by analyzing theframes in the memory. For example, for the camera feed displayed in theone or more layers, the first device 701 can analyze the framesassociated with the camera feed in the frame buffer. The one or moreobjects in FIG. 5A can be animals, vehicles, traffic signs, or the like.For example, the first device 701 can include a first layer 102 and asecond layer 104. It can be appreciated that additional layers can becontemplated extending to an n-th layer 10 n.

As shown in FIG. 5A, in a non-limiting example, the first layer 102 canbe configured to receive the feed (the received input) of the camera 106and the second layer 104 can be configured to receive the feed (thereceived input) of the camera 108. As previously mentioned, the feedfrom the camera 106 in the first layer 102 can be displayed and the feedfrom the camera 108 in the second layer 104 can be displayed. However,in an embodiment, the visibility of the first layer 102 and the secondlayer 104 can be invisible to the user, such that the first layer andthe second layer 104 can still be configured to receive the input fromthe cameras 106 and 108 but the user cannot see the feed from thecameras 106 and 108. It can be appreciated, that both feeds from thefirst layer 102 and the second layer 104 can be output to just the firstlayer 102 (or just the second layer 104) and the first device 701 candisplay the feed from just the first layer 102 (or just the second layer104).

In an embodiment, the feed from the cameras 106 and 108 can be displayedon the first layer 102 and the second layer 104 and analyzed by thefirst device 701 using computer vision to analyze, frame by frame, thefeed from the cameras 106 and 108 stored in the frame buffer. In anembodiment, the feed from the first layer 102 and the second layer 104can be analyzed by the first device 701 in the background. That is, thefirst device 701 can use computer vision to analyze, frame by frame, thefeed from the cameras 106 and 108 stored in the memory buffer of thefirst device 701. Notably, the input from the source(s) stored in thememory buffer may not be destined for displaying on the first layer 102and/or the second layer 104 and/or the n-th layer 10 n, whereas theinput from the source(s) stored in the frame buffer may typically thenbe displayed in the display of the first device 701.

Described herein, the first device 701 can calculate an entropy valuefor each of the identified objects from the sources, and the firstdevice 701 can determine a highest entropy value among calculatedentropy values for generation of a random number.

In an embodiment, in FIG. 5A, the first layer 102, the second layer 104,and all layers up to the n-th layer 10 n are layers capable of beinganalyzed by computer vision techniques via the first device 701. Thefirst layer 102 can be a layer describing the construction of explicit,meaningful descriptions of physical objects from images obtained from asource. The content in the first layer 102 can be received from a firstsource (or input), e.g., the camera 106, and the content in the secondlayer 104 can be received from a second source (or input), e.g., thecamera 108. The first layer 102 and the second layer 104 can display thereceived content (input) from the frame buffer, wherein the first layer102 and the second layer 104 can be visible, semi-visible, or invisibleto the users. In an embodiment, the first layer 102 and the second layer104 are invisible to the users and the method analyzing objects in thefirst layer 102 and the second layer 104 can be performed out of sightof the user. In an embodiment, to reduce the overall processing demandof the first device 701, the method can analyze the objects from thefirst source and the second source in the memory buffer and the objectsneed not be displayed.

The first layer 102 and the second layer 104 can be further used, by thefirst device 701, to make useful decisions about real physical objectsand scenes based on sensed images such as the one or more digital imagesfrom, but not limited to, cameras, etc. In an embodiment, the firstlayer 102 and the second layer 104 can also be video-based layers. In anembodiment, the first layer 102 may not include only content from asingle source, but it can include or receive content from more than onesource.

In an embodiment, the first layer 102 can be used for image retrieval.The algorithms for image retrieval from the computer vision techniquesanalyze the content in the query image and return results based onbest-matched content.

In an embodiment, the first layer 102 can be used for gaming andcontrols. For example, Microsoft Kinect uses stereo vision for gaming.

In an embodiment, the first layer 102 can be used for biometrics. Forexample, fingerprint, iris and face matching are some common techniquesmethods in biometric identification for use in the first layer 102.

In an embodiment, the first layer 102 can be used in smart cars capableof imaging a surrounding environment. For example, computer vision canbe used to detect traffic signs and lights and other visual featurestransmitted to the first layer 102 from one or more cameras on the smartcars used in an autonomous driving mode.

In an embodiment, the first layer 102 can be used for surveillance byreceiving input from a video recording device. For example, surveillancecameras are ubiquitous at public locations and the surveillance camerasare used to detect suspicious behaviors. The feed from the surveillancecameras can be transmitted to the device having the first layer 102, orsurveillance camera feed stored on a network video recorder (NVR) can betransmitted to the device having the first layer 102.

In an embodiment, the one or more layers receiving the input from thesources can be desktop or application layers. The first device 701 canbe a desktop computer and include a display that displays a home screenin an operating system of the first device 701. The first device 701 canalso be a user's laptop, tablet, cell phone, or the like. The firstdevice 701 can also be a device including one or more softwareapplications installed on the first device 701 such as a computer,laptop, tablet, cell phone, or the like. The first device 701 candisplay the videos or images from the sources via the first layer 102,the second layer 104, and up to the n-th layer 10 n. For example, then-th layer 10 n can be a desktop layer and displays a wallpaper imageselected by the user, and the first layer 102 can be a computervision-based application layer receiving video input from the camera 106that is overlaid on the n-th layer 10 n. The video source includes, butis not limited to, computer webcam(s), public camera(s) in variousplaces such as, but not limited to, ski resorts, train stations,airports, etc., streaming internet videos online such as, but notlimited to, Netflix, Amazon prime, YouTube, mobile device camera(s) fromusers, etc.

Again, in an embodiment, the first device 701 can include one or morelayers, wherein each layer of the one or more layers can receive inputfrom one or more sources. In an embodiment, the second layer 104 canreceive the images or videos from various cameras such as the (traffic)camera 108. The camera 108 can record vehicles, trucks, electricvehicles, or traffic lights and transmit the feed to the second layer104 of the first device 701. Thus, the user can see the images or thevideo of the vehicles, trucks, electric vehicles, or traffic lights (thesecond layer 104) overlayed on the desktop wallpaper image (the n-thlayer 10 n) of his/her first device 701.

As previously noted, the second layer 104 can also display the images orvideos from nature via the camera 106, and the first layer 102 candisplay the traffic-based images or videos from the camera 108. Thefirst layer 102 and the second layer 104 can receive the images or thevideos from a naturally occurring environment and the images or thevideos can be captured from one or more cameras, such as trafficcameras, surveillance cameras at the entrance of a national park,surveillance cameras in airports, cell phone cameras, etc.

To this end, FIG. 5B is a schematic of the various layers and theobjects received from the respective sources, according to an embodimentof the present disclosure. In an embodiment, the objects, either movingor non-moving, from the camera 106 can be displayed by the first layer102, and the objects, either moving or non-moving, from the camera 108can be displayed by the second layer 104. For example, the objects canbe a bird, a butterfly, a cloud, or the like and these can be displayedby the first layer 102. For example, the objects can be a helicopter, avehicle, an electric vehicle, a traffic light, or the like, and thesecan be displayed by the second layer 104. For simplicity, the firstlayer 102 is shown as including a first object 150, such as a butterfly,and a second object 152, such as a bird, while the second layer 104 isshown as including a third object 154, such as a car.

In an embodiment, the first device 701 can use computer visiontechniques to analyze the images or videos in the first layer 102 andthe second layer 104 and further identify the objects in the images orthe videos to determine various parameters. The parameters can be size,type, count, motion, or velocity of the objects in the images or thevideos, etc.

In an example, the first device 701 can identify the object to be abear, and the size of the bear can include a height of the bear or awidth of the bear.

In an example, the first device 701 can identify the objects to bebirds, and the count of the birds can include a number of the birds,such as 5 birds. The first device 701 can further identify the flyingdirection of the birds, the flying velocity of the birds, etc.

In an example, the first device 701 can identify the object to be avehicle, and the motion of the vehicle can include a moving direction ofthe vehicle. The first device 701 can also further identify the vehicleto have a velocity of 60 miles per hour.

In an example, the first device 701 can identify the object to be acloud in the sky and the first device 701 can further detect the movingspeed and the moving direction of the cloud in the sky.

In an embodiment, the first device 701 can identify the objects and theparameters of the objects not only in the first layer 102, but also inthe second layer 104, or in both layers 102 and 104 via the framebuffer. In an embodiment, the first device 701 can identify the objectsand the parameters of the objects via the memory buffer.

As shown in FIG. 5B, in a non-limiting example, the first device 701 canidentify the first object 150 and the second object 152 in the firstlayer 102, i.e., the bird and the butterfly, and a respective motionpath of the two objects over a period of time. As shown, the firstobject 150 can have a first motion path 150 a and the second object 152can have a second motion path 152 a. Similarly, the first device 701 canidentify the third object 154 in the second layer 104, i.e., the car,and a third motion path 154 a of the car over a period of time.

FIG. 5C is a schematic of the analyzed object motion paths converted toentropy, according to an embodiment of the present disclosure. In anembodiment, after the first device 701 analyzes the parameters of theobjects 150, 152, and 154, the calculated values of the parameters ofthe objects 150, 152, and 154 can be used by the first device 701 for adetermination of an entropy value for each of the calculated values ofthe parameters of the objects 150, 152, and 154. In an example, asshown, the motion path of the respective object can be analyzed by thefirst device 701 to determine, for example, an acceleration (a), a timeframe (t), and a direction (d) for each object 150, 152, and 154. Thefirst object 150 and the respective first motion path 150 a can beanalyzed by the first device 701 to yield acceleration (a₁), time frame(t₁), and direction (d₁) of the butterfly over the course of the firstmotion path 150 a, which can be converted to a first entropy entry 150b. It may be appreciated that while only the aforementioned parametersor variables are described as being analyzed by the first device 701,many other parameters can be analyzed and converted to the first entropyentry 150 b, some examples of which have previously been listed (e.g.,speed, size, count, etc.). Similarly, the second object 152 and therespective second motion path 152 a can yield a second entropy entry 152b and the third object 154 and the respective third motion path 154 acan yield a third entropy entry 152 b.

It may be appreciated that, as shown in FIG. 5A-5C, the objects 150,152, and 154 from the first source and the second source are shown asbeing displayed in the first layer 102 and the second layer 104 (andthus detected in the frame buffer), but the entire process can beperformed in the background of the first device 701 wherein the objectsare detected using computer vision in the memory buffer and the videostreams are not displayed in the layers at all (including displayed inthe layers with a visibility of the layers set to invisible to theuser). However, the example described with reference to FIG. 5A-5C canbe more illustrative of the determination of the object parameters, theobject entropy values, and the subsequent secure random numbergeneration. Advantageously, the example described in FIG. 5A-5C canallow the user to visually monitor the random number generation processand ensure that objects with the highest entropy are being selected togenerate the random number.

In an embodiment, the first device 701 can compare the analyzed valuesin the entropy entries 150 b, 152 b, and 154 b between one or moreobjects in the memory of the first device 701. For example, the firstdevice 701 can compare an entropy value of the first object 150, e.g.,the acceleration (a₁), with an entropy value of the second object 152,e.g., the acceleration (a₂), in the first layer 102. If the accelerationof the first object 150 is greater, then a magnitude of entropy of thefirst object 150 can be higher.

In an embodiment, the first device 701 can compare entropy valuesbetween one or more objects. For example, the first device 701 cancompare the acceleration and direction of the first object 150 in thefirst layer 102 with an acceleration and direction of the third object154 in the second layer 104. If the sum of the acceleration and thedirection of the first object 150 is greater than that of the thirdobject 154, then the magnitude of entropy of the first object 150 can behigher.

In an embodiment, a rate of change of the delta (or difference orchange) measurements between the first object 150 and the second object152 (or the third object 154) can be taken as an entropy value and canbe compared with similar delta measurements of other detected objects todetermine which of the objects has higher entropy.

In an embodiment, the first device 701 can compare an object with ahighest magnitude of entropy in the first layer 102 to another objectwith a highest magnitude of entropy in the second layer 104. Forexample, if the first object 150 has a highest magnitude of entropy inthe first layer 102 and the third object 154 has a highest magnitude ofentropy in the second layer 104, the first device 701 can compare therespective entropy entries 150 b, 154 b of these two objects 150, 154 tofind the object with a higher magnitude of entropy.

In an embodiment, a difference between entropy entries can be used tointroduce additional entropy by basing the entropy entry difference ontwo (or more) objects instead of one object, thereby increasing themagnitude of entropy for seeding the encryption algorithm. For example,a comparison between the first motion path 150 a and the second motionpath 152 a can be monitored through a predetermined time frame todetermine a difference between the first entropy entry 150 b and thesecond entropy entry 152 b. This difference inherently is based upon notjust the tracking of the butterfly (the first object 150) but also thetracking of the bird (the second object 152). Notably, these can be inthe same layer (the first layer 102) and source. Additional entropy canbe introduced by comparing objects from separate sources. For example, acomparison between the first motion path 150 a and the third motion path154 a can be monitored through a predetermined time frame to determine adifference between the first entropy entry 150 b and the third entropyentry 154 b. Again, this difference inherently is based upon not justthe tracking of the butterfly (the first object 150) but also thetracking of the car (the third object 154) which originates from anentirely separate source.

In an embodiment, the entropy entries 150 b, 152 b, and 154 b can beanalyzed for the respective magnitude of entropy and stored in anentropy entry memory 190 or log or array upon determining the magnitudeof entropy is above a predetermined threshold. For example, the car (thethird object 154) can be parked, in which case the acceleration, speed,direction, and other parameters of the car can be zero or not changingsignificantly. Thus, the first device 701 can determine the magnitude ofentropy of the car (the third object 154) is below the predeterminedthreshold and may not allow the third entropy entry 154 b to be storedor later used in seeding the encryption algorithm. The encryption canbe, for example, an AES256 encryption.

In an embodiment, the first device 701 can use a highest magnitude ofentropy to generate random numbers, as described earlier regarding theCSPRNG. The random numbers generated by this method can be unpredictableand difficult to reproduce after generation.

In an embodiment, images, videos, sources, or live streaming channelscan be accessed at time of encryption or shortly before. Identificationof the one or more objects can be performed for a threshold of time,e.g., 10 seconds, to generate the random number for the encryption. Forexample, if user A is going to send user B an email, the first device701 can predict that after user A writes the text of the email, user Ais going to hit “send”. Therefore, at the moment user A hits “NewEmail”, the first device 701 can start to obtain the live video feedmaterial in the background so as to not disrupt the email writingprocess of user A. Thus, the live video feed material can be displayedon a layer that is not visible to user A or stored in the memory buffer.The first device 701 can start detecting the objects and their metricssuch as size, type, count, motion, velocity, etc., as discussed earlier.By the time user A has entered all the text and hits “send”, the firstdevice 701 has the generated encryption key using the random number seedand encrypted the email contents, so there is no lag that user A noticesfrom hitting “send” to the email getting sent. Of course, this can alsobe performed after user A hits the “send” button, but there can be somelag which can be, for example, 10 seconds, to get the video feed and afew more seconds to analyze, detect, and encrypt.

Referring to the drawings again, FIG. 6 is a flowchart outlining amethod 500, according to an embodiment of the present disclosure. Method500 can be performed by the first device 701.

In step 505, the first device 701 can receive data (an input, a feed, astream) from a first source. For example, the first source is the camera106 and the data is a video feed from a state park.

In step 510, the first device 701 can receive data from a second source.For example, the second source is the camera 108 and the data is a videofeed from an intersection.

In step 515, the first device 701 can detect one or more objects in thefirst source and one or more objects in the second source. For example,the first device 701 can use computer vision to detect the objects.

In step 520, the first device 701 can determine one or more parametersfor the objects detected in the first source and the second source, suchas size, type, count, motion, velocity, etc. The detection of the one ormore parameters can be a continuous process.

In step 525, the first device 701 can determine entropy values for theobjects detected in the first source and the second source based on thedetermined parameter(s) for the objects.

In step 530, the first device 701 can generate a random number based onthe entropy value of each object. Notably, the random number can begenerated based on a difference between entropy entries (of high entropymagnitude) as well.

In step 535, the first device 701 can generate an encryption for databased on the generated random number. For example, the data to beencrypted can be an email. For example, the data to be encrypted can bea reference patch. That is, the encrypted data and metadata of thereference patch can be encrypted using method 500.

In an embodiment, the objects can also be any objects appearing in alive camera of a ski resort, a security camera in a city hall, etc. Inan embodiment, the objects can be from, but are not limited to, a videoof a streaming video application such as Netflix, YouTube, Amazon prime,Hulu, etc. For example, the object can be a butterfly which is currentlydisplaying in the video clip from YouTube.

In an embodiment, the one or more objects can be detected using thecomputer vision techniques and/or memory vision techniques discussedpreviously.

In an embodiment, the objects, which can be represented by a group ofone or more pixels having a specific pattern, can be detected using anyappropriate object recognition technique, such as neural networks. Theobjects to be detected for can include commonly used objects, requestsmade by users, etc.

In an embodiment, the one or more parameters can also be called visualobject observed metrics (VOOMs). For example, a size of a lion may be300 lbs. and a size of a dog may be 20 lbs., therefore, the VOOM for thelion and the dog in the video clip on YouTube can be 300 lbs and 20 lbs,respectively.

In an embodiment, the entropy values can be based on a calculation of aspecific VOOM to a full VOOM set. The calculation can also be based onan individual layer or an aggregated layer. For example, the calculationcan be only based on the VOOMs of the first layer or the second layer orthe calculation can be based on VOOMS from both layers. In anotherexample, if a VOOM of a first object in the first layer is 150, and anaverage VOOM of a set of objects in the first layer is 10, then theentropy value can be the VOOM of the first object divided by the averageVOOM, which is 15. In an embodiment, the highest entropy value from aspecific VOOM can be used as a source of entropy.

In an embodiment, the generation of the random number can be from ahighest entropy value. The entropy values can be produced in parallelamong all the layers, meaning the VOOMs of all video layers can bemeasured in parallel. For example, VOOMS of the first and the secondlayers can be, but are not limited to, the VOOM of a moving vehicle, theVOOM of a truck, the VOOM of a van, the VOOM of a cloud, the VOOM of atraffic light, the VOOM of a bird, the VOOM of a butterfly, the VOOM ofa helicopter, etc. Therefore, if the VOOM of the bird is higher than therest of the VOOMs in these two layers, the entropy value of the VOOM ofthe bird can be used for the generation of the random number.

In an embodiment, the generation of the random number can be from ahighest magnitude of entropy. The entropy values can be determined inseries from each source, and then a highest entropy value is chosen fromthe determined entropy values. For example, a first highest magnitude ofentropy and a second highest magnitude of entropy can be calculated forobjects in the first source and the second source, respectively. Aftercomparing the first and the second highest entropy magnitudes among thefirst and the second sources, a highest entropy magnitude among thesources can be determined to be used for the generation of the randomnumber.

In an embodiment, a relative entropy value of the VOOM can also be usedas an entropy source. The generation process can be a repeated processin a certain time frame. The generation can also be a continuous processamong all channels and all objects at a given time frame. The generationprocess can also be repeated until an optimized entropy value has beenfound, or an entropy value has reached a preset threshold value.

In an illustrative example using method 500, the first device 701 cangenerate the reference patch as previously described with reference toFIG. 2A-2D. The reference patch can include the unique identifier. Theunique identifier can include the encoded data that identifies thedigital content, a location address of the digital content at the seconddevice 850, a screen position within the surface area at which thedigital content is insertable in the displayed data, and the size of thedigital content when inserted in the displayed data. Notably, ingenerating the reference patch and thereby the unique identifier, thefirst device 701 can use method 500 to encrypt the encoded data of thereference patch. As the reference patch is generated, the first device701 can receive input from the camera 106 (a first source) and usingcomputer vision to detect objects from the video stream of the camera106. That is, the first device 701 can analyze, frame by frame, thevideo stream of the camera 106 in the memory buffer. As such, the securerandom number generation can occur in the background and out of sight ofthe user.

As the first device 701 detects the objects, such as a bird, a plane, aflying object, a car, and a bear, among others, the first device 701 candetermine a parameter(s) for each of the objects (size, velocity,acceleration, direction, etc.) and an entropy value for each of theobjects based on the determined parameter(s), and generate a randomnumber based on the entropy values. Upon finalizing the data to beencoded, the first device 701 can use the generated random number as aseed to encode or encrypt the data and subsequently perform the encodingor encrypting. The secured reference patch can then be embedded into thedisplayed data to be transmitted to other devices, wherein the referencepatch is secured using the encryption. The other devices can obtain thedecryption method and the seed random number in order to decode theencoded data of the reference patch when desired.

Notably, as described, the method 500 to generate the random number hasbeen described as occurring in the background of the first device 701,but it may be appreciated that the user can opt to view the entiremethod visually as the steps occur by having the sources displayed onlayers on the first device 701, wherein a visibility of the sources issufficiently visible to the user.

Embodiments of the subject matter and the functional operationsdescribed in this specification are implemented by processing circuitry(on one or more of devices 701-70 n, 850, and 1001), in tangiblyembodied computer software or firmware, in computer hardware, includingthe structures disclosed in this specification and their structuralequivalents, or in combinations of one or more of them. Embodiments ofthe subject matter described in this specification can be implemented asone or more computer programs, i.e., one or more modules of computerprogram instructions encoded on a tangible non-transitory programcarrier for execution by, or to control the operation of a dataprocessing apparatus/device, (such as the devices of FIG. 1 or thelike). The computer storage medium can be a machine-readable storagedevice, a machine-readable storage substrate, a random or serial accessmemory device, or a combination of one or more of them.

The term “data processing apparatus” refers to data processing hardwareand may encompass all kinds of apparatus, devices, and machines forprocessing data, including by way of example a programmable processor, acomputer, or multiple processors or computers. The apparatus can also beor further include special purpose logic circuitry, e.g., an FPGA (fieldprogrammable gate array) or an ASIC (application-specific integratedcircuit). The apparatus can optionally include, in addition to hardware,code that creates an execution environment for computer programs, e.g.,code that constitutes processor firmware, a protocol stack, a databasemanagement system, an operating system, or a combination of one or moreof them.

A computer program, which may also be referred to or described as aprogram, software, a software application, a module, a software module,a script, or code, can be written in any form of programming language,including compiled or interpreted languages, or declarative orprocedural languages, and it can be deployed in any form, including as astand-alone program or as a module, component, Subroutine, or other unitsuitable for use in a computing environment. A computer program may, butneed not, correspond to a file in a file system. A program can be storedin a portion of a file that holds other programs or data, e.g., one ormore scripts stored in a markup language document, in a single filededicated to the program in question, or in multiple coordinated files,e.g., files that store one or more modules, sub-programs, or portions ofcode. A computer program can be deployed to be executed on one computeror on multiple computers that are located at one site or distributedacross multiple sites and interconnected by a communication network.

The processes and logic flows described in this specification can beperformed by one or more programmable computers executing one or morecomputer programs to perform functions by operating on input data andgenerating output. The processes and logic flows can also be performedby, and apparatus can also be implemented as, special purpose logiccircuitry, e.g., an FPGA an ASIC.

Computers suitable for the execution of a computer program include, byway of example, general or special purpose microprocessors or both orany other kind of central processing unit. Generally, a CPU will receiveinstructions and data from a read-only memory or a random-access memoryor both. Elements of a computer are a CPU for performing or executinginstructions and one or more memory devices for storing instructions anddata. Generally, a computer will also include, or be operatively coupledto receive data from or transfer data to, or both, one or more massstorage devices for storing data, e.g., magnetic, magneto-optical disks,or optical disks. However, a computer need not have such devices.Moreover, a computer can be embedded in another device, e.g., a mobiletelephone, a personal digital assistant (PDA), a mobile audio or videoplayer, a game console, a Global Positioning System (GPS) receiver, or aportable storage device, e.g., a universal serial bus (USB) flash drive,to name just a few. Computer-readable media suitable for storingcomputer program instructions and data include all forms of non-volatilememory, media and memory devices, including by way of examplesemiconductor memory devices, e.g., EPROM, EEPROM, and flash memorydevices; magnetic disks, e.g., internal hard disks or removable disks;magneto optical disks; and CD-ROM and DVD-ROM disks. The processor andthe memory can be supplemented by, or incorporated in, special purposelogic circuitry.

To provide for interaction with a user, embodiments of the subjectmatter described in this specification can be implemented on a computerhaving a display device, e.g., a CRT (cathode ray tube) or LCD (liquidcrystal display) monitor, for displaying information to the user and akeyboard and a pointing device, e.g., a mouse or a trackball, by whichthe user can provide input to the computer. Other kinds of devices canbe used to provide for interaction with a user as well; for example,feedback provided to the user can be any form of sensory feedback, e.g.,visual feedback, auditory feedback, or tactile feedback; and input fromthe user can be received in any form, including acoustic, speech, ortactile input. In addition, a computer can interact with a user bysending documents to and receiving documents from a device that is usedby the user; for example, by sending web pages to a web browser on auser's device in response to requests received from the web browser.

Embodiments of the subject matter described in this specification can beimplemented in a computing system that includes a back-end component,e.g., as a data server, or that includes a middleware component, e.g.,an application server, or that includes a front-end component, e.g., aclient computer having a graphical user interface or a Web browserthrough which a user can interact with an implementation of the subjectmatter described in this specification, or any combination of one ormore Such back-end, middleware, or front-end components. The componentsof the system can be interconnected by any form or medium of digitaldata communication, e.g., a communication network. Examples ofcommunication networks include a local area network (LAN) and a widearea network (WAN), e.g., the Internet.

The computing system can include clients (user devices) and servers. Aclient and server are generally remote from each other and typicallyinteract through a communication network. The relationship of client andserver arises by virtue of computer programs running on the respectivecomputers and having a client-server relationship to each other. In anembodiment, a server transmits data, e.g., an HTML page, to a userdevice, e.g., for purposes of displaying data to and receiving userinput from a user interacting with the user device, which acts as aclient. Data generated at the user device, e.g., a result of the userinteraction, can be received from the user device at the server.

Electronic device 600 shown in FIG. 7 can be an example of one or moreof the devices shown in FIG. 1 . In an embodiment, the device 600 may bea smartphone. However, the skilled artisan will appreciate that thefeatures described herein may be adapted to be implemented on otherdevices (e.g., a laptop, a tablet, a server, an e-reader, a camera, anavigation device, etc.). The device 600 of FIG. 7 includes processingcircuitry, as discussed above. The processing circuitry includes one ormore of the elements discussed next with reference to FIG. 7 . Thedevice 600 may include other components not explicitly illustrated inFIG. 7 such as a CPU, GPU, main memory, frame buffer, etc. The device600 includes a controller 610 and a wireless communication processor 602connected to an antenna 601. A speaker 604 and a microphone 605 areconnected to a voice processor 603.

The controller 610 may include one or more processors/processingcircuitry (CPU, GPU, or other circuitry) and may control each element inthe device 600 to perform functions related to communication control,audio signal processing, graphics processing, control for the audiosignal processing, still and moving image processing and control, andother kinds of signal processing. The controller 610 may perform thesefunctions by executing instructions stored in a memory 650.Alternatively, or in addition to the local storage of the memory 650,the functions may be executed using instructions stored on an externaldevice accessed on a network or on a non-transitory computer readablemedium.

The memory 650 includes but is not limited to Read Only Memory (ROM),Random Access Memory (RAM), or a memory array including a combination ofvolatile and non-volatile memory units. The memory 650 may be utilizedas working memory by the controller 610 while executing the processesand algorithms of the present disclosure. Additionally, the memory 650may be used for long-term storage, e.g., of image data and informationrelated thereto.

The device 600 includes a control line CL and data line DL as internalcommunication bus lines. Control data to/from the controller 610 may betransmitted through the control line CL. The data line DL may be usedfor transmission of voice data, displayed data, etc.

The antenna 601 transmits/receives electromagnetic wave signals betweenbase stations for performing radio-based communication, such as thevarious forms of cellular telephone communication. The wirelesscommunication processor 602 controls the communication performed betweenthe device 600 and other external devices via the antenna 601. Forexample, the wireless communication processor 602 may controlcommunication between base stations for cellular phone communication.

The speaker 604 emits an audio signal corresponding to audio datasupplied from the voice processor 603. The microphone 605 detectssurrounding audio and converts the detected audio into an audio signal.The audio signal may then be output to the voice processor 603 forfurther processing. The voice processor 603 demodulates and/or decodesthe audio data read from the memory 650 or audio data received by thewireless communication processor 602 and/or a short-distance wirelesscommunication processor 607. Additionally, the voice processor 603 maydecode audio signals obtained by the microphone 605.

The exemplary device 600 may also include a display 620, a touch panel630, an operation key 640, and a short-distance communication processor607 connected to an antenna 606. The display 620 may be an LCD, anorganic electroluminescence display panel, or another display screentechnology. In addition to displaying still and moving image data, thedisplay 620 may display operational inputs, such as numbers or iconswhich may be used for control of the device 600. The display 620 mayadditionally display a GUI for a user to control aspects of the device600 and/or other devices. Further, the display 620 may displaycharacters and images received by the device 600 and/or stored in thememory 650 or accessed from an external device on a network. Forexample, the device 600 may access a network such as the Internet anddisplay text and/or images transmitted from a Web server.

The touch panel 630 may include a physical touch panel display screenand a touch panel driver. The touch panel 630 may include one or moretouch sensors for detecting an input operation on an operation surfaceof the touch panel display screen. The touch panel 630 also detects atouch shape and a touch area. Used herein, the phrase “touch operation”refers to an input operation performed by touching an operation surfaceof the touch panel display with an instruction object, such as a finger,thumb, or stylus-type instrument. In the case where a stylus or the likeis used in a touch operation, the stylus may include a conductivematerial at least at the tip of the stylus such that the sensorsincluded in the touch panel 630 may detect when the stylusapproaches/contacts the operation surface of the touch panel display(similar to the case in which a finger is used for the touch operation).

In certain aspects of the present disclosure, the touch panel 630 may bedisposed adjacent to the display 620 (e.g., laminated) or may be formedintegrally with the display 620. For simplicity, the present disclosureassumes the touch panel 630 is formed integrally with the display 620and therefore, examples discussed herein may describe touch operationsbeing performed on the surface of the display 620 rather than the touchpanel 630. However, the skilled artisan will appreciate that this is notlimiting.

For simplicity, the present disclosure assumes the touch panel 630 is acapacitance-type touch panel technology. However, it should beappreciated that aspects of the present disclosure may easily be appliedto other touch panel types (e.g., resistance-type touch panels) withalternate structures. In certain aspects of the present disclosure, thetouch panel 630 may include transparent electrode touch sensors arrangedin the X-Y direction on the surface of transparent sensor glass.

The touch panel driver may be included in the touch panel 630 forcontrol processing related to the touch panel 630, such as scanningcontrol. For example, the touch panel driver may scan each sensor in anelectrostatic capacitance transparent electrode pattern in theX-direction and Y-direction and detect the electrostatic capacitancevalue of each sensor to determine when a touch operation is performed.The touch panel driver may output a coordinate and correspondingelectrostatic capacitance value for each sensor. The touch panel drivermay also output a sensor identifier that may be mapped to a coordinateon the touch panel display screen. Additionally, the touch panel driverand touch panel sensors may detect when an instruction object, such as afinger is within a predetermined distance from an operation surface ofthe touch panel display screen. That is, the instruction object does notnecessarily need to directly contact the operation surface of the touchpanel display screen for touch sensors to detect the instruction objectand perform processing described herein. For example, in an embodiment,the touch panel 630 may detect a position of a user's finger around anedge of the display panel 620 (e.g., gripping a protective case thatsurrounds the display/touch panel). Signals may be transmitted by thetouch panel driver, e.g., in response to a detection of a touchoperation, in response to a query from another element based on timeddata exchange, etc.

The touch panel 630 and the display 620 may be surrounded by aprotective casing, which may also enclose the other elements included inthe device 600. In an embodiment, a position of the user's fingers onthe protective casing (but not directly on the surface of the display620) may be detected by the touch panel 630 sensors. Accordingly, thecontroller 610 may perform display control processing described hereinbased on the detected position of the user's fingers gripping thecasing. For example, an element in an interface may be moved to a newlocation within the interface (e.g., closer to one or more of thefingers) based on the detected finger position.

Further, in an embodiment, the controller 610 may be configured todetect which hand is holding the device 600, based on the detectedfinger position. For example, the touch panel 630 sensors may detect aplurality of fingers on the left side of the device 600 (e.g., on anedge of the display 620 or on the protective casing), and detect asingle finger on the right side of the device 600. In this exemplaryscenario, the controller 610 may determine that the user is holding thedevice 600 with his/her right hand because the detected grip patterncorresponds to an expected pattern when the device 600 is held only withthe right hand.

The operation key 640 may include one or more buttons or similarexternal control elements, which may generate an operation signal basedon a detected input by the user. In addition to outputs from the touchpanel 630, these operation signals may be supplied to the controller 610for performing related processing and control. In certain aspects of thepresent disclosure, the processing and/or functions associated withexternal buttons and the like may be performed by the controller 610 inresponse to an input operation on the touch panel 630 display screenrather than the external button, key, etc. In this way, external buttonson the device 600 may be eliminated in lieu of performing inputs viatouch operations, thereby improving watertightness.

The antenna 606 may transmit/receive electromagnetic wave signalsto/from other external apparatuses, and the short-distance wirelesscommunication processor 607 may control the wireless communicationperformed between the other external apparatuses. Bluetooth, IEEE802.11, and near-field communication (NFC) are non-limiting examples ofwireless communication protocols that may be used for inter-devicecommunication via the short-distance wireless communication processor607.

The device 600 may include a motion sensor 608. The motion sensor 608may detect features of motion (i.e., one or more movements) of thedevice 600. For example, the motion sensor 608 may include anaccelerometer to detect acceleration, a gyroscope to detect angularvelocity, a geomagnetic sensor to detect direction, a geo-locationsensor to detect location, etc., or a combination thereof to detectmotion of the device 600. In an embodiment, the motion sensor 608 maygenerate a detection signal that includes data representing the detectedmotion. For example, the motion sensor 608 may determine a number ofdistinct movements in a motion (e.g., from start of the series ofmovements to the stop, within a predetermined time interval, etc.), anumber of physical shocks on the device 600 (e.g., a jarring, hitting,etc., of the electronic device), a speed and/or acceleration of themotion (instantaneous and/or temporal), or other motion features. Thedetected motion features may be included in the generated detectionsignal. The detection signal may be transmitted, e.g., to the controller610, whereby further processing may be performed based on data includedin the detection signal. The motion sensor 608 can work in conjunctionwith a Global Positioning System (GPS) section 660. The information ofthe present position detected by the GPS section 660 is transmitted tothe controller 610. An antenna 661 is connected to the GPS section 660for receiving and transmitting signals to and from a GPS satellite.

The device 600 may include a camera section 609, which includes a lensand shutter for capturing photographs of the surroundings around thedevice 600. In an embodiment, the camera section 609 capturessurroundings of an opposite side of the device 600 from the user. Theimages of the captured photographs can be displayed on the display panel620. A memory section saves the captured photographs. The memory sectionmay reside within the camera section 609 or it may be part of the memory650. The camera section 609 can be a separate feature attached to thedevice 600 or it can be a built-in camera feature.

An example of a type of computer is shown in FIG. 8 . The computer 700can be used for the operations described in association with any of thecomputer-implement methods described previously, according to oneimplementation. For example, the computer 700 can be an example ofdevices 701, 702, 70 n, 1001, or a server (such as device 850). Thecomputer 700 includes processing circuitry, as discussed above. Thedevice 850 may include other components not explicitly illustrated inFIG. 8 such as a CPU, GPU, main memory, frame buffer, etc. Theprocessing circuitry includes one or more of the elements discussed nextwith reference to FIG. 8 . In FIG. 8 , the computer 700 includes aprocessor 710, a memory 720, a storage device 730, and an input/outputdevice 740. Each of the components 710, 720, 730, and 740 areinterconnected using a system bus 750. The processor 710 is capable ofprocessing instructions for execution within the system 700. In oneimplementation, the processor 710 is a single-threaded processor. Inanother implementation, the processor 710 is a multi-threaded processor.The processor 710 is capable of processing instructions stored in thememory 720 or on the storage device 730 to display graphical informationfor a user interface on the input/output device 740.

The memory 720 stores information within the computer 700. In oneimplementation, the memory 720 is a computer-readable medium. In oneimplementation, the memory 720 is a volatile memory. In anotherimplementation, the memory 720 is a non-volatile memory.

The storage device 730 is capable of providing mass storage for thesystem 700. In one implementation, the storage device 730 is acomputer-readable medium. In various different implementations, thestorage device 730 may be a floppy disk device, a hard disk device, anoptical disk device, or a tape device.

The input/output device 740 provides input/output operations for thecomputer 700. In one implementation, the input/output device 740includes a keyboard and/or pointing device. In another implementation,the input/output device 740 includes a display for displaying graphicaluser interfaces.

Next, a hardware description of a device 801 according to exemplaryembodiments is described with reference to FIG. 9 . In FIG. 9 , thedevice 801, which can be the above described devices of FIG. 1 ,includes processing circuitry, as discussed above. The processingcircuitry includes one or more of the elements discussed next withreference to FIG. 9 . The device 1201 may include other components notexplicitly illustrated in FIG. 9 such as a CPU, GPU, main memory, framebuffer, etc. In FIG. 9 , the device 801 includes a CPU 800 whichperforms the processes described above/below. The process data andinstructions may be stored in memory 802. These processes andinstructions may also be stored on a storage medium disk 804 such as ahard drive (HDD) or portable storage medium or may be stored remotely.Further, the claimed advancements are not limited by the form of thecomputer-readable media on which the instructions of the inventiveprocess are stored. For example, the instructions may be stored on CDs,DVDs, in FLASH memory, RAM, ROM, PROM, EPROM, EEPROM, hard disk or anyother information processing device with which the device communicates,such as a server or computer.

Further, the claimed advancements may be provided as a utilityapplication, background daemon, or component of an operating system, orcombination thereof, executing in conjunction with CPU 800 and anoperating system such as Microsoft Windows, UNIX, Solaris, LINUX, AppleMAC-OS and other systems known to those skilled in the art.

The hardware elements in order to achieve the device may be realized byvarious circuitry elements, known to those skilled in the art. Forexample, CPU 800 may be a Xenon or Core processor from Intel of Americaor an Opteron processor from AMD of America, or may be other processortypes that would be recognized by one of ordinary skill in the art.Alternatively, the CPU 800 may be implemented on an FPGA, ASIC, PLD orusing discrete logic circuits, as one of ordinary skill in the art wouldrecognize. Further, CPU 800 may be implemented as multiple processorscooperatively working in parallel to perform the instructions of theprocesses described above. CPU 800 can be an example of the CPUillustrated in each of the devices of FIG. 1 .

The device 801 in FIG. 9 also includes a network controller 806, such asan Intel Ethernet PRO network interface card from Intel Corporation ofAmerica, for interfacing with the network 851 (also shown in FIG. 1 ),and to communicate with the other devices of FIG. 1 . As can beappreciated, the network 851 can be a public network, such as theInternet, or a private network such as an LAN or WAN network, or anycombination thereof and can also include PSTN or ISDN sub-networks. Thenetwork 851 can also be wired, such as an Ethernet network, or can bewireless such as a cellular network including EDGE, 3G, 4G and 5Gwireless cellular systems. The wireless network can also be WiFi,Bluetooth, or any other wireless form of communication that is known.

The device further includes a display controller 808, such as a NVIDIAGeForce GTX or Quadro graphics adaptor from NVIDIA Corporation ofAmerica for interfacing with display 810, such as an LCD monitor. Ageneral purpose I/O interface 812 interfaces with a keyboard and/ormouse 814 as well as a touch screen panel 816 on or separate fromdisplay 810. General purpose I/O interface also connects to a variety ofperipherals 818 including printers and scanners.

A sound controller 820 is also provided in the device to interface withspeakers/microphone 822 thereby providing sounds and/or music.

The general-purpose storage controller 824 connects the storage mediumdisk 804 with communication bus 826, which may be an ISA, EISA, VESA,PCI, or similar, for interconnecting all of the components of thedevice. A description of the general features and functionality of thedisplay 810, keyboard and/or mouse 814, as well as the displaycontroller 808, storage controller 824, network controller 806, soundcontroller 820, and general purpose I/O interface 812 is omitted hereinfor brevity as these features are known.

As shown in FIG. 10 , in some embodiments, one or more of the disclosedfunctions and capabilities may be used to enable a volumetric compositeof content-activated layers of Transparent Computing, content-agnosticlayers of Transparent Computing and/or camera-captured layers ofTransparent Computing placed visibly behind 2-dimensional or3-dimensional content displayed on screens, placed in front of2-dimensional or 3-dimensional content displayed on screens, placedinside of 3-dimensional content displayed on screens and/or placedvirtually outside of the display of screens. Users can interact viaTouchless Computing with any layer in a volumetric composite of layersof Transparent Computing wherein a user's gaze, gestures, movements,position, orientation, or other characteristics observed by a camera areused as the basis for selecting and interacting with objects in anylayer in the volumetric composite of layers of Transparent Computing toexecute processes on computing devices.

In some embodiments, one or more of the disclosed functions andcapabilities may be used to enable users to see a volumetric compositeof layers of Transparent Computing from a 360-degree Optical LenticularPerspective wherein a user's gaze, gestures, movements, position,orientation, or other characteristics observed by cameras are a basis tocalculate, derive and/or predict the 360-degree Optical LenticularPerspective from which users see the volumetric composite of layers ofTransparent Computing displayed on screens. Further, users can engagewith a 3-dimensional virtual environment displayed on screens consistingof layers of Transparent Computing placed behind the 3-dimensionalvirtual environment displayed on screens, placed in front of a3-dimensional virtual environment displayed on screens, and/or placedinside of the a 3-dimensional virtual environment displayed on screenswherein users can select and interact with objects in any layer ofTransparent Computing to execute processes on computing devices whilelooking at the combination of the 3-dimensional virtual environment andthe volumetric composite of layers of Transparent Computing from anyangle of the 360-degree Optical Lenticular Perspective available tousers.

Obviously, numerous modifications and variations are possible in lightof the above teachings. It is therefore to be understood that within thescope of the appended claims, embodiments of the present disclosure maybe practiced otherwise than as specifically described herein.

Embodiments of the present disclosure may also be as set forth in thefollowing description.

An apparatus, including: processing circuitry configured to access aframe buffer of the GPU, analyze, in the frame buffer of the GPU, afirst frame representing a first section of a stream of displayed datafrom a source, the first frame being displayed by the device, the sourcebeing an image or video-based feed including an object, based on theanalyzed first frame, identify the object in the first frame from thesource, the object associated with a parameter having a first valuecorresponding to the first frame of the source, analyze, in the framebuffer of the GPU, a second frame representing a second section of thestream of the displayed data from the source that is being displayed bythe device, based on the analyzed second frame, identify the object inthe second frame from the source, the object associated with a secondvalue of the parameter corresponding to the second frame, determine anentropy of the object based on a difference between the first value ofthe parameter of the object corresponding to the first frame of thesource and the second value of the parameter corresponding to the secondframe of the source, the entropy corresponding to a degree of randomnessof the object, generate a random number based on the entropy of theobject, and generate an encryption for encrypting data based on thegenerated random number.

In an embodiment, the parameter includes a size of the object, adirection of the object, a velocity of the object, an acceleration ofthe object, or a quantity of the object.

In an embodiment, the processing circuitry is further configured toidentify the object in the first frame of the source by determining theparameter and a second parameter of the object in the first frame of thesource, the second parameter having a first value corresponding to thefirst frame of the source, identify the object in the second frame ofthe source by determining the parameter and the second parameter of theobject in the second frame of the source, the second parameter having asecond value corresponding to the second frame of the source, anddetermine an entropy of the object based on i) a difference between thefirst value of the parameter of the object corresponding to the firstframe of the source and the second value of the parameter correspondingto the second frame of the source, and ii) a difference between thefirst value of the second parameter of the object corresponding to thefirst frame of the source and the second value of the second parametercorresponding to the second frame of the source.

In an embodiment, the processing circuitry is further configured toanalyze, in the frame buffer of the GPU, a first frame representing afirst section of a second stream of displayed data from a second sourcethat is being displayed by the device, the second source being an imageor video-based feed including a second object, based on the analyzedfirst frame of the second source, identify the second object in thefirst frame of the second source, the second object including theparameter having a first value corresponding to the first frame of thesecond source, analyze, in the frame buffer of the GPU, a second framerepresenting a second section of the stream of displayed data from thesecond source that is being displayed by the device, based on theanalyzed second frame of the second source, identify the second objectin the second frame of the second source, the second objecting includinga second value of the parameter corresponding to the second frame of thesecond source, and determine an entropy of the second object based on adifference between the first value of the parameter of the second objectcorresponding to the first frame of the second source and the secondvalue of the parameter of the second object corresponding to the secondframe of the second source.

In an embodiment, the processing circuitry is further configured togenerate the random number based on a difference between the entropy ofthe object and the entropy of the second object.

In an embodiment, the processing circuitry is further configured togenerate the random number based on a combination of the entropy of theobject and the entropy of the second object.

In an embodiment, the processing circuitry is further configured todetermine a highest entropy between the entropy of the object and theentropy of the second object, and generate the random number based onthe highest entropy and the corresponding object.

A method, including: accessing a frame buffer of a graphics processingunit (GPU); analyzing, in the frame buffer of the GPU, a first framerepresenting a first section of a stream of displayed data from asource, the first frame being displayed by a device, the source being animage or video-based feed including an object; based on the analyzedfirst frame, identifying the object in the first frame from the source,the object associated with a parameter having a first valuecorresponding to the first frame of the source; analyzing, in the framebuffer of the GPU, a second frame representing a second section of thestream of the displayed data from the source that is being displayed bythe device; based on the analyzed second frame, identifying the objectin the second frame from the source, the object associated with a secondvalue of the parameter corresponding to the second frame; determining anentropy of the object based on a difference between the first value ofthe parameter of the object corresponding to the first frame of thesource and the second value of the parameter corresponding to the secondframe of the source, the entropy corresponding to a degree of randomnessof the object; generating a random number based on the entropy of theobject; and generating an encryption for encrypting data based on thegenerated random number.

In an embodiment, the parameter includes a size of the object, adirection of the object, a velocity of the object, an acceleration ofthe object, or a quantity of the object.

In an embodiment, the method further comprises identifying the object inthe first frame of the source by determining the parameter and a secondparameter of the object in the first frame of the source, the secondparameter having a first value corresponding to the first frame of thesource; identifying the object in the second frame of the source bydetermining the parameter and the second parameter of the object in thesecond frame of the source, the second parameter having a second valuecorresponding to the second frame of the source; and determining anentropy of the object based on i) a difference between the first valueof the parameter of the object corresponding to the first frame of thesource and the second value of the parameter corresponding to the secondframe of the source, and ii) a difference between the first value of thesecond parameter of the object corresponding to the first frame of thesource and the second value of the second parameter corresponding to thesecond frame of the source.

In an embodiment, the method further comprises analyzing, in the framebuffer of the GPU, a first frame representing a first section of asecond stream of displayed data from a second source that is beingdisplayed by the device, the second source being an image or video-basedfeed including a second object; based on the analyzed first frame of thesecond source, identifying the second object in the first frame of thesecond source, the second object including the parameter having a firstvalue corresponding to the first frame of the second source; analyzing,in the frame buffer of the GPU, a second frame representing a secondsection of the stream of displayed data from the second source that isbeing displayed by the device; based on the analyzed second frame of thesecond source, identifying the second object in the second frame of thesecond source, the second objecting including a second value of theparameter corresponding to the second frame of the second source; anddetermining an entropy of the second object based on a differencebetween the first value of the parameter of the second objectcorresponding to the first frame of the second source and the secondvalue of the parameter of the second object corresponding to the secondframe of the second source.

In an embodiment, the method further comprises generating the randomnumber based on a difference between the entropy of the object and theentropy of the second object.

In an embodiment, the method further comprises generating the randomnumber based on a combination of the entropy of the object and theentropy of the second object.

In an embodiment, the method further comprises determining a highestentropy between the entropy of the object and the entropy of the secondobject; and generating the random number based on the highest entropyand the corresponding object.

A non-transitory computer-readable storage medium for storingcomputer-readable instructions that, when executed by a computer, causethe computer to perform a method, the method comprising: accessing aframe buffer of a graphics processing unit (GPU); analyzing, in theframe buffer of the GPU, a first frame representing a first section of astream of displayed data from a source, the first frame being displayedby a device, the source being an image or video-based feed including anobject; based on the analyzed first frame, identifying the object in thefirst frame from the source, the object associated with a parameterhaving a first value corresponding to the first frame of the source;analyzing, in the frame buffer of the GPU, a second frame representing asecond section of the stream of the displayed data from the source thatis being displayed by the device; based on the analyzed second frame,identifying the object in the second frame from the source, the objectassociated with a second value of the parameter corresponding to thesecond frame; determining an entropy of the object based on a differencebetween the first value of the parameter of the object corresponding tothe first frame of the source and the second value of the parametercorresponding to the second frame of the source, the entropycorresponding to a degree of randomness of the object; generating arandom number based on the entropy of the object; and generating anencryption for encrypting data based on the generated random number.

In an embodiment, the parameter includes a size of the object, adirection of the object, a velocity of the object, an acceleration ofthe object, or a quantity of the object.

In an embodiment, the method performed by the computer further comprisesidentifying the object in the first frame of the source by determiningthe parameter and a second parameter of the object in the first frame ofthe source, the second parameter having a first value corresponding tothe first frame of the source; identifying the object in the secondframe of the source by determining the parameter and the secondparameter of the object in the second frame of the source, the secondparameter having a second value corresponding to the second frame of thesource; and determining an entropy of the object based on i) adifference between the first value of the parameter of the objectcorresponding to the first frame of the source and the second value ofthe parameter corresponding to the second frame of the source, and ii) adifference between the first value of the second parameter of the objectcorresponding to the first frame of the source and the second value ofthe second parameter corresponding to the second frame of the source.

In an embodiment, the method performed by the computer further comprisesidentifying the object in the first frame of the source by determiningthe parameter and a second parameter of the object in the first frame ofthe source, the second parameter having a first value corresponding tothe first frame of the source; identifying the object in the secondframe of the source by determining the parameter and the secondparameter of the object in the second frame of the source, the secondparameter having a second value corresponding to the second frame of thesource; and determining an entropy of the object based on i) adifference between the first value of the parameter of the objectcorresponding to the first frame of the source and the second value ofthe parameter corresponding to the second frame of the source, and ii) adifference between the first value of the second parameter of the objectcorresponding to the first frame of the source and the second value ofthe second parameter corresponding to the second frame of the source.

In an embodiment, the method performed by the computer further comprisesanalyzing, in the frame buffer of the GPU, a first frame representing afirst section of a second stream of displayed data from a second sourcethat is being displayed by the device, the second source being an imageor video-based feed including a second object; based on the analyzedfirst frame of the second source, identifying the second object in thefirst frame of the second source, the second object including theparameter having a first value corresponding to the first frame of thesecond source; analyzing, in the frame buffer of the GPU, a second framerepresenting a second section of the stream of displayed data from thesecond source that is being displayed by the device; based on theanalyzed second frame of the second source, identifying the secondobject in the second frame of the second source, the second objectingincluding a second value of the parameter corresponding to the secondframe of the second source; and determining an entropy of the secondobject based on a difference between the first value of the parameter ofthe second object corresponding to the first frame of the second sourceand the second value of the parameter of the second object correspondingto the second frame of the second source.

In an embodiment, the method performed by the computer further comprisesgenerating the random number based on a difference between the entropyof the object and the entropy of the second object.

In an embodiment, the method further comprises generating the randomnumber based on a combination of the entropy of the object and theentropy of the second object.

While this specification contains many specific implementation details,these should not be construed as limitations on the scope of what may beclaimed, but rather as descriptions of features that may be specific toparticular embodiments.

Certain features that are described in this specification in the contextof separate embodiments can also be implemented in combination in asingle embodiment. Conversely, various features that are described inthe context of a single embodiment can also be implemented in multipleembodiments separately or in any suitable sub-combination. Moreover,although features may be described above as acting in certaincombinations and even initially claimed as such, one or more featuresfrom a claimed combination can in some cases be excised from thecombination, and the claimed combination may be directed to asub-combination or variation of a sub-combination.

Similarly, while operations are depicted in the drawings in a particularorder, this should not be understood as requiring that such operationsbe performed in the particular order shown or in sequential order, orthat all illustrated operations be performed, to achieve desirableresults. In certain circumstances, multitasking and parallel processingmay be advantageous. Moreover, the separation of various system modulesand components in the embodiments described above should not beunderstood as requiring such separation in all embodiments, and itshould be understood that the described program components and systemscan generally be integrated together in a single software product orpackaged into multiple software products.

Particular embodiments of the subject matter have been described. Otherembodiments are within the scope of the following claims. For example,the actions recited in the claims can be performed in a different orderand still achieve desirable results. As one example, the processesdepicted in the accompanying figures do not necessarily require theparticular order shown, or sequential order, to achieve desirableresults. In some cases, multitasking and parallel processing may beadvantageous.

Thus, the foregoing discussion discloses and describes merely exemplaryembodiments of the present disclosure. As will be understood by thoseskilled in the art, the present disclosure may be embodied in otherspecific forms without departing from the spirit thereof. Accordingly,the disclosure of the present disclosure is intended to be illustrative,but not limiting of the scope of the disclosure, as well as otherclaims. The disclosure, including any readily discernible variants ofthe teachings herein, defines, in part, the scope of the foregoing claimterminology such that no inventive subject matter is dedicated to thepublic.

1. A device, comprising: processing circuitry configured to identify ina frame buffer an object in a first frame representing a first sectionof a stream of displayed data from a source, the object associated witha parameter having a first value corresponding to the first frame of thesource, identify in the frame buffer the object in a second framerepresenting a second section of the stream of the displayed data fromthe source, the object associated with a second value of the parametercorresponding to the second frame, generate a random number based on adifference between the first value of the parameter of the objectcorresponding to the first frame of the source and the second value ofthe parameter corresponding to the second frame of the source.
 2. Thedevice of claim 1, wherein an entropy of the object corresponds to adegree of randomness of the object and the entropy represents thedifference between the first value of the parameter of the objectcorresponding to the first frame of the source and the second value ofthe parameter corresponding to the second frame of the source.
 3. Thedevice of claim 1, wherein the parameter includes a size of the object,a direction of the object, a velocity of the object, an acceleration ofthe object, or a quantity of the object.
 4. The device of claim 1,wherein the processing circuitry is further configured to identify theobject in the first frame of the source by determining the parameter anda second parameter of the object in the first frame of the source, thesecond parameter having a first value corresponding to the first frameof the source, identify the object in the second frame of the source bydetermining the parameter and the second parameter of the object in thesecond frame of the source, the second parameter having a second valuecorresponding to the second frame of the source, and determine anentropy of the object based on i) a difference between the first valueof the parameter of the object corresponding to the first frame of thesource and the second value of the parameter corresponding to the secondframe of the source, and ii) a difference between the first value of thesecond parameter of the object corresponding to the first frame of thesource and the second value of the second parameter corresponding to thesecond frame of the source.
 5. The device of claim 1, wherein theprocessing circuitry is further configured to identify a second objectin a first frame representing a first section of a second stream ofdisplayed data from a second source, the second object including theparameter having a first value corresponding to the first frame of thesecond source, identify the second object in a second frame representinga second section of the stream of displayed data from the second source,and determine an entropy of the second object based on a differencebetween the first value of the parameter of the second objectcorresponding to the first frame of the second source and the secondvalue of the parameter of the second object corresponding to the secondframe of the second source.
 6. The device of claim 5, wherein theprocessing circuitry is further configured to generate the random numberbased on a difference between an entropy of the object and the entropyof the second object.
 7. The device of claim 5, wherein the processingcircuitry is further configured to generate the random number based on acombination of an entropy of the object and the entropy of the secondobject.
 8. The device of claim 5, wherein the processing circuitry isfurther configured to determine a highest entropy between an entropy ofthe object and the entropy of the second object, and generate the randomnumber based on the highest entropy and the corresponding object.
 9. Thedevice of claim 5, wherein the second object includes a second value ofthe parameter corresponding to the second frame of the second source.10. A method, comprising: identifying in a frame buffer an object in afirst frame representing a first section of a stream of displayed datafrom a source, the object associated with a parameter having a firstvalue corresponding to the first frame of the source; identifying in theframe buffer the object in a second frame representing a second sectionof the stream of the displayed data from the source, the objectassociated with a second value of the parameter corresponding to thesecond frame; generating a random number based on a difference betweenthe first value of the parameter of the object corresponding to thefirst frame of the source and the second value of the parametercorresponding to the second frame of the source.
 11. The method of claim10, wherein an entropy of the object corresponds to a degree ofrandomness of the object and the entropy represents the differencebetween the first value of the parameter of the object corresponding tothe first frame of the source and the second value of the parametercorresponding to the second frame of the source.
 12. The method of claim10, wherein the parameter includes a size of the object, a direction ofthe object, a velocity of the object, an acceleration of the object, ora quantity of the object.
 13. The method of claim 10, wherein theidentifying identifies the object in the first frame of the source bydetermining the parameter and a second parameter of the object in thefirst frame of the source, the second parameter having a first valuecorresponding to the first frame of the source, the identifyingidentifies the object in the second frame of the source by determiningthe parameter and the second parameter of the object in the second frameof the source, the second parameter having a second value correspondingto the second frame of the source, and the determining determines anentropy of the object based on i) a difference between the first valueof the parameter of the object corresponding to the first frame of thesource and the second value of the parameter corresponding to the secondframe of the source, and ii) a difference between the first value of thesecond parameter of the object corresponding to the first frame of thesource and the second value of the second parameter corresponding to thesecond frame of the source.
 14. The method of claim 10, wherein theidentifying identifies a second object in a first frame representing afirst section of a second stream of displayed data from a second source,the second object including the parameter having a first valuecorresponding to the first frame of the second source, the identifyingidentifies the second object in a second frame representing a secondsection of the stream of displayed data from the second source, and thedetermining determines an entropy of the second object based on adifference between the first value of the parameter of the second objectcorresponding to the first frame of the second source and the secondvalue of the parameter of the second object corresponding to the secondframe of the second source.
 15. The method of claim 14, furthercomprising generating the random number based on a difference between anentropy of the object and the entropy of the second object.
 16. Themethod of claim 14, further comprising generating the random numberbased on a combination of an entropy of the object and the entropy ofthe second object.
 17. The method of claim 14, further comprisingdetermining a highest entropy between an entropy of the object and theentropy of the second object, wherein the generating generates therandom number based on the highest entropy and the corresponding object.18. The method of claim 14, wherein the second object includes a secondvalue of the parameter corresponding to the second frame of the secondsource.
 19. A non-transitory computer-readable storage medium forstoring computer-readable instructions that, when executed by acomputer, cause the computer to perform a method, the method comprising:identifying in a frame buffer an object in a first frame representing afirst section of a stream of displayed data from a source, the objectassociated with a parameter having a first value corresponding to thefirst frame of the source; identifying in the frame buffer the object ina second frame representing a second section of the stream of thedisplayed data from the source, the object associated with a secondvalue of the parameter corresponding to the second frame; determining anentropy of the object based on a difference between the first value ofthe parameter of the object corresponding to the first frame of thesource and the second value of the parameter corresponding to the secondframe of the source, the entropy corresponding to a degree of randomnessof the object; and generating a random number based on the entropy ofthe object.
 20. The non-transitory computer-readable storage mediumaccording to claim 19, wherein the source is an image or video-basedfeed.